How to Understand Data Streaming in Terms of Python

Here is an example that shows data streaming and loading data at once. In Anaconda’s local directory, you can create a text file. Add some data to it. You can use it to understand the concept.

IN THIS PAGE

  1. Getting Data at once into Memory
  2. Python logic to get data at once to memory
  3. Commands used
  4. Streaming data

Getting Data at once into Memory

When you take the entire data into memory, your program may hang if your system lacks memory. It then causes you to restart the process is a time-consuming task. So the point you need to note here is loading data at once may throw performance issues.

A sample file
A sample file

Python logic to get data at once to memory

The sample.py script has python code. Its functionality is to open the file in read mode and get the data at once. This is the best example to tell in interviews. I am giving this here for your understanding.

Python logic to read the file
Python logic to read the file

Commands used

  • nano sample.py
  • ./sample.py
Python script's output
Python script’s output

Streaming data

Here I have used for-loop and get records one by one. The method, you can say data streaming. The stream.py has for-loop logic added. Which reads records (data) one by one.

The stream.py details
The stream.py details

The output you can see is displayed the file records one by one.

Streaming data Python logic with for-loop
Streaming data Python logic with for-loop

With the above example, you have got a clear idea of streaming. It’s helpful for data science engineers who use Python for analysis.

Related

Author: Srini

Experienced software developer. Skills in Development, Coding, Testing and Debugging. Good Data analytic skills (Data Warehousing and BI). Also skills in Mainframe.