Datacorn

@midudev 👇 x.com/datacorn_io/st…

¡Pedazo de biblioteca de Atlassian para Drag & Drop! ✓ Funciona en React, Vue, Angular y Svelte ✓ Usada por Trello, Jira y Confluence ✓ Tamaño pequeño: 4.7KB ✓ Compatible con móviles → npm install @atlaskit/pragmatic-drag-and-drop

@alexxubyte 👇 x.com/datacorn_io/st…

Almost every software engineer has used Linux before, but only a handful know how its Boot Process works :) Let's dive in.

@sahnlam x.com/datacorn_io/st…

Load Balancer Basics Load balancers are essential components in modern application architectures, designed to distribute incoming traffic efficiently across multiple servers. Load balancers improve application performance, availability, and scalability. Traffic Distribution: Load balancers evenly distribute incoming traffic among a pool of servers, ensuring optimal resource utilization and preventing any single server from becoming overwhelmed. Algorithms like round-robin or least connections are used to select the most suitable server for each request. High Availability: If a server fails, the load balancer automatically redirects traffic to the remaining healthy servers. This ensures that the application remains accessible even in the event of server failures, minimizing downtime and improving overall availability. SSL Termination: Load balancers can handle SSL/TLS encryption and decryption, offloading this CPU-intensive task from backend servers. This improves server performance and simplifies SSL certificate management. Session Persistence: For applications that require maintaining user sessions on a specific server, load balancers support session persistence. They ensure that subsequent requests from a user are consistently routed to the same server, preserving session integrity. Scalability: Load balancers facilitate horizontal scaling by allowing easy addition of servers to the pool. As traffic increases, new servers can be provisioned, and the load balancer will automatically distribute the load across all servers, enabling seamless scalability. Health Monitoring: Load balancers continuously monitor server health and performance. They exclude unhealthy servers from the pool, ensuring that only healthy servers handle incoming requests. This proactive monitoring maintains optimal application performance. – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/496keA7

@techyoutbe D Service

Q. Which Kubernetes object is responsible for managing load balancing and routing traffic to a set of pods? A. Deployment B. DaemonSet C. Ingress D. Service

@techyoutbe D, terraform import

Q. Which command is used to import existing infrastructure into Terraform? A. terraform state import B. terraform apply C. terraform init D. terraform import

@techyoutbe Both

Terraform or Ansible?

In machine learning, synthetic data can offer real performance improvements #SyntheticData #MachineLearning #AI #LLaMa news.mit.edu/2022/synthetic…

@llama_index @databricks @Data_AI_Summit Thank you

Building an Advanced Research Agent on @databricks 📈 We gave a 90 minute workshop last week at the @Data_AI_Summit on building an advanced research assistant with two focus areas beyond the naive RAG setup. 1. Improving Data Quality: Ensuring good parsing, chunking, indexing modules that can handle complex data types. LlamaParse is well-suited for this. 2. Improving Query Complexity: Adding layers of agentic reasoning to progressively handle more sophisticated queries over your data. We’re excited to share the full set of slides and two workshop notebooks below. We use LLMs available through Databricks and local @huggingface embeddings - you can easily run this within your own Databricks environment. Full slides: docs.google.com/presentation/d… [Improving Data Quality] Building a financial RAG pipeline with LlamaParse: colab.research.google.com/drive/18RUkf8I… [Improving Query Complexity] Building an assistant over research papers: colab.research.google.com/drive/18RUkf8I…

@ezekiel_aleke Very interesting, thank you very much. We will take it into account

Dear Data Analyst! Learn the LookUps NOW. Repost for others

@freest_man Very interesting, thank you very much. We will take it into account

ETL vs. ELT: What's the difference? Let's understand with business Examples (This might be asked in your DS interview) Before that, you can understand what Extraction, Transformation and Loading are in this Tweet: x.com/freest_man/sta… ETL Extract, transform, and load (ETL) is a data pipeline used to collect data from various sources. It then transforms the data according to business rules and loads the data into a destination data store. The transformation work in ETL takes place in a specialized engine, and it often involves using staging tables to temporarily hold data as it is being transformed and ultimately loaded to its destination. ELT Extract, load, and transform (ELT) differs from ETL solely in where the transformation takes place. In the ELT pipeline, the transformation occurs in the target data store. Instead of using a separate transformation engine, the processing capabilities of the target data store are used to transform data. This simplifies the architecture by removing the transformation engine from the pipeline. Examples of ETL/ELT ETL: Your organization has started to explore more about historical trends and patterns of the data. Currently, the organization only has a transactional database (OLTP) for the product, and running these heavy analytical queries runs the risk of breaking the transactional database for the product. Thus, you decide to replicate the data on a database designed specifically for analytics (OLAP) to power these heavy queries while not risking the production transactions database. You build an ETL pipeline to replicate data from the transactional database to the analytical database, including some data transformations to make it easier to use. ELT: Your organization purchases third-party data from a vendor to supplement your organization's data. While this third-party data is useful, the vendor provides extremely messy tables (Which is usually the scenario). Since this data is for R&D purposes, there is no defined business logic available. Therefore, you deem it’s okay to dump this data into a data lake and give the data science team access to explore. You build an ELT pipeline that extracts the raw data from the third-party vendor and loads it into the data lake. After a few iterations, the data science team determines which parts of the data are valuable, and builds data transformations on top of the data in the data lake for their workflows.

@Python_Dv Very interesting, thank you very much. We will take it into account

Python Tip Don't Forget To Like ♥️ | Share 📲 | Comment 💬 | Save 📥 #python #programming #developer #programmer #coding #coder #softwaredeveloper #computerscience #webdev #webdeveloper #webdevelopment #pythonprogramming #pythonquiz #ai #ml #machinelearning #datascience

@DataScienceDojo Very interesting, thank you very much. We will take it into account

Ever wonder how raw data gets transformed into the insights that power AI and data science? Its data pipeline! Enroll in our exclusive 5-day LLM Bootcamp ➡️hubs.la/Q02C69Bl0 Join our Discord server ➡️hubs.la/Q02C697q0 #DataPipeline #Datascience #LLMs

@learnk8s Very interesting, thank you very much. We will take it into account

This article discusses the complexities of learning Kubernetes and suggests that understanding its API is the most straightforward approach The article also provides a guide on how to manipulate the API ➜ @talhakhalid101/simplest-way-to-learn-kubernetes-is-with-its-api-07e1fb8c6e0f" target="_blank" rel="nofollow noopener">medium.com/@talhakhalid10…

@ProfTomYeh Very interesting, thank you very much. We will take it into account

[Superposition] by Hand✍️ Superposition is a key property that differentiates a qubit in a quantum computer from a bit in a classical computer. A bit is always in one position. A qubit can be in many positions simultaneously, until it is measured. This “super” power to be in many positions at once is what enables certain quantum algorithms to achieve exponential speedup! How can we calculate superposition by hand? [1] Given ↳ 3 Qubits: 🟦 a, 🟧 b and 🟪 c. ↳ A quantum circuit with 5 operations involving these 3 qubits. ↳ Because each quantum operation creates two branches, we will show how this circuit is exploring 2 ^ 5 = 32 positions simultaneously. [2] 🟦 Set a ↳ Qubit a is simultaneously exploring two new branches: |0⟩ and |1⟩ ↳ a = |0⟩ means we want a to have a positive chance in |0⟩ and no chance in |1⟩. We write + in |0⟩ and o in |1⟩ to indicate this. ↳ Now, the quantum circuit is exploring 2 positions simultaneously. [3] 🟧 Set b ↳ Continuing each branch from the previous step, Qubit b is simultaneously exploring two new branches: |0⟩ and |1⟩. ↳ b = |1⟩ means we want b to have no chance in |0⟩ and a positive chance in |1⟩. We write o in |0⟩ and 1 in |1⟩ to indicate this. ↳ Now, the quantum circuit is exploring 2 x 2 = 4 positions simultaneously. [4] 🟦 H|a⟩ Hadamard Gate ↳ Continuing each branch from the previous step, Qubit a is simultaneously exploring two new branches: |0⟩ and |1⟩ by applying an Hadamard gate. ↳ The main purpose of the H gate is to create “superposition” using the following rules: + |0⟩ → + |0⟩ + |1⟩ + |1⟩ → + |0⟩ - |1⟩ ↳ Since a has + in |0⟩, the result is +’s for both |0⟩ and |1⟩. ↳ Intuitively, it is like split the + in the |0⟩ branch into two +’s, one for |0⟩ and one for |1⟩. ↳ Now, the quantum circuit is exploring 2 x 2 x 2 = 8 positions simultaneously. [5] 🟧 H|b⟩ Hadamard Gate ↳ Continuing each branch from the previous step, Qubit b is simultaneously exploring two new branches: |0⟩ and |1⟩ by applying an Hadamard gate. ↳ Recall the rules are: + |0⟩ → + |0⟩ + |1⟩ + |1⟩ → + |0⟩ - |1⟩ ↳ Since b has + in |1⟩, the result is + for |0⟩ and - for |1⟩. ↳ Now, the quantum circuit is exploring 2 x 2 x 2 x 2 = 16 positions simultaneously. [6] 🟪 Set c ↳ Qubit c is simultaneously exploring two new branches: |0⟩ and |1⟩ ↳ c = |0⟩ means we want c to have a positive chance in |0⟩ and no chance in |1⟩. We write + in |0⟩ and o in |1⟩ to indicate this. ↳ Now, the quantum circuit is exploring 2 x 2 x 2 x 2 x 2 = 32 positions simultaneously. [7] Possible Paths ↳ Even though this quantum circuit is exploring all 32 probable paths simultaneously, only some paths are possible. ↳ If a branch has a 0, it means it is not possible to take the path through the branch. ↳ In other words, if a path has + or - for all the branches along the way, it is possible. ↳ We found only four possible paths without any zero along the way: 000 010 100 110 [8] Quantum Measurement ↳ When taking a measurement, the quantum nature will randomly choose one of the possibilities. ↳ Since there are four possibilities, each possibility has an equal 25% chance of being measured. ↳ Once measured, they collapse into just one of these possibilities. Notes: Mathematically, the quantum circuit in this exercise can be calculated using the Dirac notation as follows: H|0⟩ ⊗ H|1⟩ ⊗ |0⟩ = (1/2) * ( |000⟩ + |010⟩ + |100⟩ + |110⟩) In developing this exercise, my goal is to invent a simpler and more accessible method to calculate quantum circuits by hand without resorting to difficult math such as Dirac notation, complex numbers, and tensor-product, while yielding the same results. I would appreciate your feedback. Your feedback will help me develop more exercises like this on other quantum computing topics such as entanglement, control gates, and quantum algorithms like Deutsch–Jozsa, Shor, Bernstein–Vazirani, and Glover.

@milan_milanovic Very interesting, thank you very much. We will take it into account

𝗬𝗼𝘂𝗿 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗜𝘀 𝗔𝗹𝘄𝗮𝘆𝘀 𝗖𝗼𝗺𝗽𝗹𝗲𝘅 𝗔𝘀 𝗬𝗼𝘂𝗿 𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 Have you heard about 𝗖𝗼𝗻𝘄𝗮𝘆'𝘀 𝗟𝗮𝘄? It is a theory created by computer scientist Melvin Conway in 1967. which says: "𝘖𝘳𝘨𝘢𝘯𝘪𝘻𝘢𝘵𝘪𝘰𝘯𝘴, 𝘸𝘩𝘰 𝘥𝘦𝘴𝘪𝘨𝘯 𝘴𝘺𝘴𝘵𝘦𝘮𝘴, 𝘢𝘳𝘦 𝘤𝘰𝘯𝘴𝘵𝘳𝘢𝘪𝘯𝘦𝘥 𝘵𝘰 𝘱𝘳𝘰𝘥𝘶𝘤𝘦 𝘥𝘦𝘴𝘪𝘨𝘯𝘴 𝘸𝘩𝘪𝘤𝘩 𝘢𝘳𝘦 𝘤𝘰𝘱𝘪𝘦𝘴 𝘰𝘧 𝘵𝘩𝘦 𝘤𝘰𝘮𝘮𝘶𝘯𝘪𝘤𝘢𝘵𝘪𝘰𝘯 𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦𝘴 𝘰𝘧 𝘵𝘩𝘦𝘴𝘦 𝘰𝘳𝘨𝘢𝘯𝘪𝘻𝘢𝘵𝘪𝘰𝘯𝘴." In other words, the 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗼𝗳 𝗮 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝗶𝘀 𝗼𝗳𝘁𝗲𝗻 𝗶𝗻𝗳𝗹𝘂𝗲𝗻𝗰𝗲𝗱 𝗯𝘆 𝘁𝗵𝗲 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝘂𝗻𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝘁𝗲𝗮𝗺 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗶𝘁. This can result in more optimal software architecture for the problem being solved, as the team may focus on their own organizational needs over the system's needs. This means 𝗮𝗻 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝘀𝗺𝗮𝗹𝗹 𝗱𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝗱 𝘁𝗲𝗮𝗺𝘀 𝘄𝗶𝗹𝗹 𝗽𝗿𝗼𝗱𝘂𝗰𝗲 𝗮 𝗺𝗼𝗱𝘂𝗹𝗮𝗿 𝘀𝗲𝗿𝘃𝗶𝗰𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲, 𝘄𝗵𝗶𝗹𝗲 𝗮𝗻 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗹𝗮𝗿𝗴𝗲 𝗰𝗼𝗹𝗹𝗼𝗰𝗮𝘁𝗲𝗱 𝘁𝗲𝗮𝗺𝘀 𝘄𝗶𝗹𝗹 𝗽𝗿𝗼𝗱𝘂𝗰𝗲 𝗮 𝗺𝗼𝗻𝗼𝗹𝗶𝘁𝗵𝗶𝗰 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲. In some broad sense, we could even say that 𝗛𝗥 𝘂𝘀𝘂𝗮𝗹𝗹𝘆 𝗱𝗲𝗳𝗶𝗻𝗲𝘀 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀. To mitigate this, we can use the 𝗜𝗻𝘃𝗲𝗿𝘀𝗲 𝗖𝗼𝗻𝘄𝗮𝘆 𝗺𝗮𝗻𝗲𝘂𝘃𝗲𝗿. This technique means we should involve software architects, engineers, and leaders in defining organizational structures. Doing it can lead to better software. Yet, we can see that many organizations ignore Conway's law and think that organizational structures and software architecture are detached from each other, with surprises in the end. Back to you, what are your experiences with Conway's Law? Image credits: Manu Cornet (bonkersworld .net). #softwarearchitecture

@parmardarshil07 Very interesting, thank you very much. We will take it into account

Becoming AWS Data Engineer (Quick Guide) 📄 There are 100s of services available on Amazon Web Services, the thing is you don't need to learn all of them. You need to focus on 10-20 services that you might use as a Data Engineer and services for other work (networking/access management) 📈 AWS Services for Data Engineers: ✅ Simple Storage Service (S3): It is an object storage (you can store anything) basically, the center of your work, all of the data you will get will be stored here for future processing. ✅ AWS Glue: Want to write ETL (Extract, Transform, Load) jobs in Python/Spark without worrying about servers? Glue is a serverless service where you just need to focus on writing your code and everything will be taken care of by AWS ✅ Amazon Redshift: So you processed data and wrote the ETL script, where to load it? Yes! Data Warehouse. Amazon Redshift is a fully managed data warehouse service that makes it simple to analyze all your data using standard SQL and your preferred business intelligence (BI) tools. ✅ Amazon EMR (Elastic MapReduce): Maybe you already have your processing scripts on on-premise servers written in Hadoop/Spark and want to migrate to a similar system then EMR is the way to go. It is a managed big data platform that makes it easy to process large amounts of data using open-source data processing frameworks like Apache Hadoop and Apache Spark. ✅ AWS Lambda: Want to run quick scripts on specific times/triggers/events? Lambda is your best friend! AWS Lambda is a serverless computing service that enables you to run your code without managing servers. This service can be used to run data engineering workflows and transform data in real-time. ✅ Amazon Athena: Why load data on Data Warehouse, when you can just direct run SQL query on top of actual files? Athena is an ad-hoc query interface that you can use for SQL queries directly on top of files stored on S3 buckets ✅ Kinesis: Want to process real-time data, analyze it and store it? Kinesis is used for this type of work, just like Apache Kafka, you can process real-time data using Kinesis. ✅ DMS (Data Migration Service): I spent my early data engineering days working on migration projects and DMS makes your life so much easier. There are many more services you can focus on such as EC2, IAM, VPC, Batch, Sagemaker, etc... If you want to learn all of these services by building a project then check out the comments 👇🏻

What we have learned from talking with 100+ data scientists The main challenges data science teams face nowadays and the communication gap with executives #datascience #machinelearning #dataquality #bigdata #artificialintelligence medium.com/ydata-ai/what-…

Datasets for computer vision: the true catalyst for progress #computervision #machinevision #datasetsforml #syntheticdata #deeplearning #artificialintelligence #machinelearning #datascience medium.com/anyverse/datas…

@LevelUpCoding_ Thank you

Database Indexing Explained Most databases require some form of indexing to keep up with performance benchmarks. Searching through a database is much simpler when the data is correctly indexed, which improves the system's overall performance. A database index is a lot like the index on the back of a book. It saves you time and energy by allowing you to easily find what you're looking for without having to flick through every page. Database indexes work the same way. An index is a key-value pair where the key is used to search for data instead of the corresponding indexed column(s), and the value is a pointer to the relevant row(s) in the table. To get the most out of your database, you should use the right index type for the job. The B-tree is one of the most commonly used indexing structures where keys are hierarchically sorted. When searching data, the tree is traversed down to the leaf node that contains the appropriate key and pointer to the relevant rows in the table. B-tree is most commonly used because of its efficiency in storing and searching through ordered data. Their balanced structure means that all keys can be accessed in the same number of steps, making performance consistent. Hash indexes are best used when you are searching for an exact value match. The key component of a hash index is the hash function. When searching for a specific value, the search value is passed through a hash function which returns a hash value. That hash value tells the database where the key and pointers are located in the hash table. Bitmap indexing is used for columns with few unique values. Each bitmap represents a unique value. A bitmap indicates the presence or absence of a value in a dataset, using 1’s & 0’s. For existing values, the position of the 1 in the bitmap shows the location of the row in the table. Bitmap indexes are very effective in handling complex queries where multiple columns are used. When you are indexing a table, make sure to carefully select the columns to be indexed based on the most frequently used columns in WHERE clauses. A composite index may be used when multiple columns are often used in a WHERE clause together. With a composite index, a combination of two or more columns are used to create a concatenated key. The keys are then stored based on the index strategy, such as the options mentioned above. Indexing can be a double-edged sword. It significantly speeds up queries, but it also takes up storage space and adds overhead to operations. Balancing performance & optimal storage is crucial to get the most out of your database without introducing inefficiencies.