Azik

devs

Originally built for log processing, Kafka now powers a wide range of applications. Its durable message storage and flexible data access enables consumers to pull records at their convenience.

Here are some popular Kafka use cases:

- 𝗟𝗼𝗴 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀: Efficiently handles massive volumes of log data for analysis and insight generation.

- 𝗗𝗮𝘁𝗮 𝗦𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗳𝗼𝗿 𝗥𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝘀: Powers real-time data streaming to deliver personalized recommendations.

- 𝗦𝘆𝘀𝘁𝗲𝗺 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗹𝗲𝗿𝘁𝗶𝗻𝗴: Facilitates real-time monitoring and alerting systems for timely responses to system metrics.

- 𝗖𝗵𝗮𝗻𝗴𝗲 𝗗𝗮𝘁𝗮 𝗖𝗮𝗽𝘁𝘂𝗿𝗲 (CDC): Captures and processes database changes to keep data in sync across systems.

- 𝗦𝘆𝘀𝘁𝗲𝗺 𝗠𝗶𝗴𝗿𝗮𝘁𝗶𝗼𝗻: Supports the seamless migration of systems by ensuring data consistency and availability.

![](https://m.stacker.news/30309)


Top 5 Kafka Use Cases

Push notifications keep users engaged, but setting up a solid system can be quite the challenge. We're talking in-app notifications, emails, SMS, and even social media pushes – it's a lot to handle!

So, how does it all work?

🔹 First off, the business services send notifications to the gateway. It can handle them one by one or in batches.

🔹 Next, the distribution service takes over. It validates, formats, and schedules the messages based on user preferences and templates.

🔹 Then, the notifications head to the routers (usually message queues) for smooth delivery.

🔹 The channel services take it from there, making sure the notifications reach users through various channels – in-app, email, SMS, social media, you name it!

🔹 Finally, the tracking and analytics service keeps an eye on the delivery metrics. This way, the operations team can check out the reports and make the user experience even better.

It's a complex process that we often take for granted.

![](https://m.stacker.news/30203)

Push Notification System: The Basics

Encoding, encryption, and tokenization are three distinct processes that handle data in different ways for various purposes, including data transmission, security, and compliance. In system design, it's important to choose the right approach to handle sensitive information.

🔹 Encoding
Encoding converts data into a different format using a reversible scheme. For example, Base64 encoding changes binary data into ASCII characters, allowing it to be processed by text-based systems.

Encoding is not meant to secure data. Encoded data can be easily decoded using the same scheme without a key.

🔹 Encryption
Encryption transforms data into a secure format using algorithms and keys. Symmetric encryption uses the same key for both encryption and decryption, while asymmetric encryption uses a public key for encryption and a private key for decryption.

Encryption protects data confidentiality by converting readable data (plaintext) into an unreadable format (ciphertext). Only those with the correct key can decrypt and access the original data.

🔹 Tokenization
Tokenization replaces sensitive data with non-sensitive placeholders called tokens. The mapping between the original data and the token is securely stored in a token vault. Tokens can be used across various systems without revealing the original data, reducing the risk of data breaches.

Tokenization is often used to protect credit card information, personal identification numbers, and other sensitive data. It's secure because tokens do not contain any part of the original data and cannot be reverse-engineered. Tokenization is particularly useful for compliance with regulations like PCI DSS.

![](https://m.stacker.news/29961)

Encoding vs. Encryption vs. Tokenization

Hypertext Transfer Protocol Secure (HTTPS) is an extension of HTTP that utilizes Transport Layer Security (TLS) to encrypt communication between a client and server.  Any intercepted data will be unreadable and secure from tampering and eavesdropping.

What's the process for encrypting and decrypting data?

Step 1 - The journey begins with the client (like your browser) establishing a TCP connection with the server.

Step 2 - Next comes the “client hello” where the browser sends a message containing supported cipher suites and the highest TLS version it can handle. Cipher suites are sets of algorithms that typically include: a key exchange method to share keys between devices, a bulk encryption algorithm to encrypt data, and a message authentication code algorithm to check data integrity.

The server responds with a “server hello”, confirming the chosen cipher suite and TLS version that they can both understand. The server then sends a TLS certificate to the client containing its domain name, certificate authority signature, and the server’s public key. The client checks this certificate to validate it is trusted and belongs to the server.

Step 3 - Once the TLS certificate is validated, the client creates a session key to be used for encrypting the bulk data transfer. Bulk data transfer refers to the transmission of the actual application data between client and server once the secure TLS connection is established. To securely send this session key to the server, it’s encrypted with the server’s public key. The server, with its private key, is the only one who can decrypt this encrypted session key.

Step 4 - Now that both parties have the secret session key, they shift gears to symmetric encryption. It’s like they’ve agreed on a private language that only they understand. This makes the data transfer very secure. Symmetric encryption is much faster for large amounts of data.

![IMG_20240503_184935.jpg](https://m.stacker.news/29411)

How does HTTPS work?

PayPal runs 1.5K brokers worldwide and hosts over 20K topics.

A recent post on their engineering blog explains how they did it:

1. Cluster Management: Instead of having fixed addresses for their brokers, PayPal created a service that dynamically handles these address

2. Access Control Lists (ACLs): PayPal set up a system to ensure only authorized applications could access specific Kafka clusters and topics, enhancing security.

3. Monitoring and Alerting: PayPal gets a lot of data on how well their Kafka clusters work to ensure they are reliable. An alert is sent out if something strange happens, making it easier for the team to deal with the problem immediately.

4. QA Environment: PayPal built a separate replica world (QA environment) where developers can freely test things without affecting the real-world setup.

5. Topic Onboarding: Teams must make requests to start new topics. Before approving these requests, PayPal has a Kafka team that looks at them, makes sure they can handle them and sets up any security steps that are needed.

![IMG_20240502_230853.jpg](https://m.stacker.news/29309)

How can Kafka be scaled so it can handle 1+ trillion daily messages?

Dogs_And_Cats

![20240502_150901.jpg](https://m.stacker.news/29280)

every time my dog eats he is just very excited.

Symmetric encryption and asymmetric encryption are two types of cryptographic techniques used to secure data and communications, but they differ in their methods of encryption and decryption.

🔹 In symmetric encryption, a single key is used for both the encryption and decryption of data. It is faster and can be applied to bulk data encryption/decryption. For example, we can use it to encrypt large amounts of PII (personally identifiable information). It poses challenges in key management because the sender and receiver share the same key.

🔹 Asymmetric encryption uses a pair of keys: a public key and a private key. The public key is freely distributed and used to encrypt data, while the private key is kept secret and used to decrypt the data. It is more secure than symmetric encryption because the private key is never shared. However, asymmetric encryption is slower because of the complexity of key generation and mathematical computations. For example, HTTPS uses asymmetric encryption to exchange session keys during the TLS handshake, and after that, HTTPS uses symmetric encryption for subsequent communications.

![IMG_20240501_135809.png](https://m.stacker.news/29128)

Symmetric vs Asymmetric Encryption

What are its principles, methods, constraints, and best practices? 

I hope the diagram below gives you a quick overview.

![20240501_014649.jpg](https://m.stacker.news/29039)

How does REST API work? 

How C++, Java, and Python Work Under the Hood?

Compiled languages like C++ and Go transform source code into machine code using a compiler before execution. The compiled machine code can then be directly executed by the CPU.

Java first compiles source code into bytecode, which is platform-independent and executed by the Java Virtual Machine (JVM). The JVM can further boost performance using Just-In-Time (JIT) compilation to convert bytecode to machine code at runtime.

Interpreted languages like Javascript and Ruby don't undergo compilation. Instead, their code is processed line-by-line by an interpreter during execution. However, modern JavaScript engines like V8 also utilize JIT compilation for enhanced performance.

Python is a mix of both worlds. It first compiles source code into platform-independent bytecode, which is then executed line-by-line by the platform-dependent interpreter. In addition, implementations like PyPy use JIT compilation for a speed boost.

Generally, compiled languages can offer speed advantages, but the line between compiled and interpreted languages is increasingly blurred thanks to modern optimization techniques.

![IMG_20240430_144319.jpg](https://m.stacker.news/28905)

IBM MQ -> RabbitMQ -> Kafka -> Pulsar

𝗜𝗕𝗠 𝗠𝗤
IBM MQ was launched in 1993. It was renamed to WebSphere MQ in 2002 and to IBM MQ in 2014. IBM MQ is a widely successful product, used extensively in the financial sector.

𝗥𝗮𝗯𝗯𝗶𝘁𝗠𝗤
RabbitMQ's architecture differs from IBM MQ and more closely resembles Kafka. Producers publish messages to exchanges with specified exchange types like direct, topic, or fanout. Exchanges then route and deliver messages to queues based on predefined bindings and rules. Consumers subscribe to queues and receive messages routed to them by the exchanges.

𝗞𝗮𝗳𝗸𝗮
In 2011, LinkedIn open-sourced Kafka, a distributed event streaming platform. Kafka optimizes for write throughput and handles real-time data feeds with low latency. Its unified event log enables event streaming. Its simplicity and fault tolerance have allowed it to replace products like AMQP-based queues.

𝗣𝘂𝗹𝘀𝗮𝗿
Originally developed by Yahoo, Pulsar is an all-in-one messaging and streaming platform. Its cloud-native architecture enables better cluster scaling and partition migration. Pulsar uses a two-layer architecture with a serving layer and persistent layer. It natively supports tiered storage, leveraging cheaper object storage like AWS S3 for long-term persistence.

Which message queues have you used?

![IMG_20240430_143158.png](https://m.stacker.news/28901)

The Evolution of Message Queues

Modern web-scale applications require specialized databases to handle massive amounts of data. Relational databases alone are often not enough.

𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝗮𝗹 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 optimize complex queries across large datasets using column-oriented storage and aggregation. This enables interactive analysis of billions of rows.

𝗡𝗼𝗦𝗤𝗟 databases provide flexibility beyond the tabular relational structure:

- 𝗞𝗲𝘆-𝘃𝗮𝗹𝘂𝗲 𝘀𝘁𝗼𝗿𝗲𝘀 enable fast, simple data access.
- 𝗗𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 store hierarchical JSON data. Highly scalable.
- 𝗖𝗼𝗹𝘂𝗺𝗻𝗮𝗿 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲𝘀 optimize analytics via column-based structure.
- 𝗚𝗿𝗮𝗽𝗵 databases efficiently store interconnected data.

Choosing the right database to match the access patterns and data models is key. A typical web application may use a mix of relational, NoSQL, and other databases to deliver top performance at scale.

![IMG_20240429_111345.jpg](https://m.stacker.news/28703)

Managing Data at Scale

Every Apartment needs this Upgrade! (WhatsApp/Telegram Notifications)

security

Hacker attack invades Lula Government’s payment system!

The Federal Government's Integrated Financial Administration System (Siafi) platform was the target of criminality acting in the virtual environment.  According to investigations by the Federal Police, there is a suspicion that a hacker attack may have embezzled public money.

![20240422_204948.jpg](https://m.stacker.news/27750)

This is Jon Johansen, the Norwegian hacker who broke the DVD security code

Stranded or on some island with a wallet full of Bitcoin

giving rise to mass piracy.

![20240421_210639.jpg](https://m.stacker.news/27579)

![20240420_103044.jpg](https://m.stacker.news/27421)

Difference Between

bitdevs

![20240419_191705.jpg](https://m.stacker.news/27361)

The halving

𝗞𝗲𝘆 𝗣𝗹𝗮𝘆𝗲𝗿𝘀 𝗶𝗻 𝗦𝗦𝗢

1. User - The individual seeking access to applications
2. Identity Provider (IDP) - Authenticates users (e.g. Google, Facebook)
3. Application - The software or service the user wants to access

𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝗼𝗳 𝗦𝗦𝗢

- Simplifies access with one set of credentials
- Enhances user experience
- Reduces password fatigue
- Centralizes security and access management
- Improves security
- Streamlines access control
- Simplifies compliance reporting
- Allows seamless integration

𝗛𝗼𝘄 𝗦𝗦𝗢 𝗪𝗼𝗿𝗸𝘀

- User tries to log into an application
- Application redirects user to SSO/IDP
- IDP authenticates user
- IDP issues authenticated token
- Token sent back to browser
- Browser presents token to application
- Application grants access without re-entering credentials

This streamlined process enables single login access to multiple applications, improving convenience and security.

Over to you: With SSO facilitating access across various platforms, what measures do you consider essential to maintain the integrity and security of user identities?

![IMG_20240419_134616.jpg](https://m.stacker.news/27341)



Single Sign-On (SSO) explained in simple terms.

This diagram provides a high-level roadmap of the core SQL concepts and how they fit together.

It covers key elements like databases, tables, data types, functions, and essential statements to query and modify data.

Use this visual overview to guide your learning journey and grasp how the different pieces of SQL work together.

![IMG_20240419_133715.jpg](https://m.stacker.news/27339)