OpenObserve for Enterprises on LinkedIn: #sre #openobserveforenterprises #observability #datavisualization #sql (2024)

💽Detect and Analyze Inefficient Database Access Patterns with Dynatrace💽As I am preparing for next weeks "What isDynatraceand how to get started" I am sharing a second common problem pattern around #inefficient #database calls, e.g: Too many queries, slow queries, failing queries, queries returning too much data.🤓Watch my video here on LinkedIn where I walk through this Use Case!🖊️Sign up and join me on Tue, Aug 9th for my live session:https://lnkd.in/dxnD-u7B#observability#distributedsystems#opentelemetry #database#dynatrace#howto

42

For anyone working to build online aggregation solutions can explore timescaleDB. been digging this one for a while and definitely seems like a good option. It has a custom storage mechanism for the event store , use Gorilla compression (redis and prom uses same) , auto partitions your data and query plans always gives u good clues about the amount of data / partition scanned.Feel free to explorehttps://lnkd.in/gV6z9m3X#dataengineering

11

Heading to #RSAC this year? Aflac gooses its data efficiency with Cribl:Context is king–that’s why optimizing your data, enriching it in the stream, and having the ability to see it and tweak it before sending it into analytics tools or storage can be a game changer. When you are reviewing or correlating the data to troubleshoot, run investigations and respond, think how much time your team can save by having the right geolocation, asset, timestamp and even threat intel already associated with the log data! 👉Aflac Set up and used a Redis cache along with Cribl Stream to enhance their data before sending it to its destination. 👉Imported a 34 million row CSV file into Redis and used the Redis function to match fields to records to add a new field that is used for faster identification of data once it is in Splunk.#rsa #rsa24 #casestudy #dataoptimization #mttr

6

Got my attention: The advantages of queues on logs (https://lnkd.in/gHqYx8US)

How to Reduce License Usage in SplunkLicense Usage Can be reduced, by doing...--> Data Filtering --> Index Time Field Extraction1) Data Filtering--> Filter out unnecessary data while onboarding the data.--> Use configuration files to drop or modify incoming data.props.conf[source::/var/log/messages]TRANSFORMS-null= setnulltransforms.conf[setnull]REGEX = \[sshd\]DEST_KEY = queueFORMAT = nullQueue2) Index Time Field Extraction--> Extract only essential fields--> This reduces the volume of raw data being indexed in Splunk.Note: You should store only the extracted fields, and replace _raw field value with the fields extracted.(e.g: If there are 25 words in raw data, this action is storing only 10 words, the license consumption will be reduced.)#splunk #splunkadmin #splunkblogs #splunkengineer #splunklife #softmania #splunkmania

60

12 Comments

Explore Advanced SPL Commands! 💡 Enhance your data analysis skills and extract deeper insights with these advanced Splunk SPL commands. #Splunk #DataAnalysis #DataScience #TechSkills"appendThe append command merges events from two or more datasets into a single result set.Syntax: ... | append [| ]Example: search index=main | append [| search index=secondary]👾👾👾👾👾👾👾👾👾👾👾👾appendcolsThe appendcols command appends fields from a subsearch to each event in the main search.Syntax: ... | appendcols [ | ]Example: search index=main | appendcols [ | stats count as event_count | fields event_count ]👾👾👾👾👾👾👾👾👾👾👾👾spathThe spath command extracts fields from XML or JSON data.Syntax: ... | spath <XML/JSON_field>Example: search index=weblogs | spath user_agent👾👾👾👾👾👾👾👾👾👾👾👾xmlkvThe xmlkv command extracts key-value pairs from XML data.Syntax: ... | xmlkv <XML_field>Example: search index=weblogs | xmlkv message👾👾👾👾👾👾👾👾👾👾👾👾mvcombineThe mvcombine command combines multivalue fields into a single multivalue field.Syntax: ... | mvcombine <multivalue_field>Example: search index=weblogs | mvcombine src_ip👾👾👾👾👾👾👾👾👾👾👾👾mvexpandThe mvexpand command expands multivalue fields into separate events for each value.Syntax: ... | mvexpand <multivalue_field>Example: search index=weblogs | mvexpand src_ip👾👾👾👾👾👾👾👾👾👾👾👾streamstatsThe streamstats command calculates statistics on streaming data and adds the results as fields to each event.Syntax: ... | streamstats <function(field)> byExample: search status=200 | streamstats count by host...https://lnkd.in/d52MncBa

8

In August 2023, #Datadog released #FlexLogs – a lower-cost, warm storage tier for log data.While we applaud Datadog for providing a lower-cost option to support log data, there are two major drawbacks to Flex Logs.1️⃣ Query Performance vs. Cost TradeoffWith Flex Logs, there is an implicit tradeoff: sacrifice performance to reduce storage costs. And to incrementally improve performance via different compute options, you give up cost savings.2️⃣ What Data Goes in What Tier?To realize the value of Flex Logs, you must first classify your log data by storage tier: Standard (hot) vs. Flex (warm) vs. Archive (cold). Datadog provides no in-application assistance, and very few observability teams know this information off the top of their heads, nor do they have the spare cycles to figure it out.In our latest blog post, we discuss the solution in greater detail. If you’re considering Flex Logs, we suggest you give it a read. 👇 The link is in our comments 👇

26

2 Comments

You may use ELK/EFK or Splunk to aggregate logs from distributed nodes and analyze logs.But I think, the following need to be considered to choose a solution.- sql queriable: all the logs need to be queried using sql in standard way.- cost-effective: logs need to be saved to cost-effective storage such as table format iceberg.- joining multiple databases: joining multiple databases need to be supported to richer analyze logs.Here is a story about Analyzing logs using Chango Log which is a log agent to read log files and send logs to Chango.- https://lnkd.in/eD-d-VdD

2

Here are some of the key concepts one should be familiar with before the system design interview: - Caching- Storage- Replication- Data partitioning techniques- Load balancing- Performance measurement of scalable systems.- API design- Scalability, availability, and reliability of large systems.- Forward and reverse proxies.- SQL vs. NoSQL Feel free to add more topics to the list. #systemdesign #hld #techinterview

66

1 Comment

There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.Read more 👉 https://aikn.co/cdde56#Phrase #Performance #MalformedData #Cloudflare