User Tools

Site Tools

Trace:
task_boiler

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
task_boiler [2022/08/11 21:56] – [Practical solution] rajutask_boiler [2024/01/22 22:59] – [Issue] raju
Line 4: Line 4:
 Start the first heading with "=====" Start the first heading with "====="
  
-===== Apply unix commands to all but the first line ===== +===== blocked by barrons ===== 
-==== Situation ==== +==== Issue ==== 
-Let's say we want to sort a series of numbers in descending order but keep the header at the top. For example, given+barrons.com is giving the following error.
 <code> <code>
-$ echo -e "value\n8\n2\n6\n3" +You have been blocked.
-value +
-8 +
-2 +
-6 +
-3 +
-</code> +
-we want to output +
-<code> +
-value +
-2 +
-3 +
-6 +
-8 +
-</code>+
  
-We can't directly use 'sort' since it will sort the header as well. +Why? 
-<code> +Something about the behaviour of the browser has caught our 
-$ echo -e "value\n8\n2\n6\n3" | sort +attention.
-2 +
-3 +
-6 +
-8 +
-value +
-</code>+
  
-==== Bare bones solution ==== +There are various possible explanations for this: 
-Create a script called 'body' with the following contents +* You are browsing and clicking at a speed much faster than 
-<code> +  expected of a human being. 
-#!/usr/bin/env bash +* Something is preventing JavaScript from working on your 
-# +  computer. 
-# body: apply expression to all but the first line. +* There is a robot on the same network (IP WW.XX.YY.ZZ) as you.
-# Use multiple times in case the header spans more than one line. +
-# +
-# Example usage: +
-# $ echo -e "value\n8\n2\n6\n3" | body sort +
-# +
-IFS= read -r header +
-printf '%s\n' "$header" +
-"$@" +
-</code>+
  
-Make it executable +Having problems accessing the site? Submit feedback. 
-<code> +ID: a-b-c-d-e
-chmod +x body+
 </code> </code>
  
-place it somewhere in your PATH (say ~/bin) +==== Solution ==== 
-<code> +Click on the extensions button in the top right 
-mv body ~/bin +-> Bypass Paywalls Clean -> settings button -> Options 
-</code>+-> scroll down to "BPC settings" section 
 +-> check "Barron's - no Googlebot (http error 500)" option 
 +-> Save
  
-This script will apply any unix command to all but the first line. For example, using it on our example +==== System Information ==== 
- +  * bypass paywalls clean extension version: 3.5.2.1 (latest asof 2024-01-22) 
-<code> +  * google chrome: 120.0.6099.200 
-$ echo -e "value\n8\n2\n6\n3" | body sort +  * OS: Windows 10 Enterprise
-value +
-2 +
-3 +
-6 +
-8 +
-</code> +
- +
-==== Practical solution ==== +
-I got the above script from https://github.com/jeroenjanssens/dsutils/blob/master/body . The underlying repository (https://github.com/jeroenjanssens/dsutils) contains many such useful scripts (ex:- header - to add, replace, and delete header lines). +
- +
-A more practical approach is to clone that repository and add it to your PATH. +
- +
-I did it as follows. +
- +
-Remove the bare bones script we added in the previous step +
-<code> +
-% rm ~/bin/body +
-</code> +
- +
-Clone the repository +
-<code> +
-% mkdir -p ~/github/jeroenjanssens +
-% cd ~/github/jeroenjanssens +
- +
-% git clone git@github.com:jeroenjanssens/dsutils.git +
-Cloning into 'dsutils'... +
-remote: Enumerating objects: 62, done. +
-remote: Counting objects: 100% (62/62), done. +
-remote: Compressing objects: 100% (52/52), done. +
-remote: Total 62 (delta 20), reused 48 (delta 10), pack-reused 0 +
-Receiving objects: 100% (62/62), 18.59 KiB | 9.29 MiB/s, done. +
-Resolving deltas: 100% (20/20), done. +
-</code> +
- +
-Update the PATH in ~/.bashrc by adding the following lines +
-<code> +
-#------------------------------------------------------------------------------ +
-# Add data science utils such as body, header +
-export PATH=~/github/jeroenjanssens/dsutils:$PATH +
-#------------------------------------------------------------------------------ +
-</code> +
- +
-Open a new bash session and verify that these utilities are correctly picked up. +
-<code> +
-% which body +
-/home/rajulocal/github/jeroenjanssens/dsutils/body +
- +
-% which header +
-/home/rajulocal/github/jeroenjanssens/dsutils/header +
-</code> +
- +
-Verify that the utilities are working as expected. +
-<code> +
-% echo -e "value\n8\n2\n6\n3" | body sort +
-value +
-2 +
-3 +
-6 +
-8 +
-</code> +
- +
-==== Works with any command ==== +
-The beauty of this approach is that you can use it with any unix command (and not just sort). For example, you can grep a value and it will show the header along with the value. +
-<code> +
-% echo -e "value\n8\n2\n6\n3" | body grep 8 +
-value +
-8 +
-</code>+
  
-==== How I came across it ==== 
-I came across it while reading the book "Data Science at the Command Line" (2nd Edition) by Jeroen Janssens (https://smile.amazon.com/Data-Science-Command-Line-Explore/dp/1492087912). The book is available for free at https://datascienceatthecommandline.com/2e/ . The 'body' and 'header' commands are discussed in https://datascienceatthecommandline.com/2e/chapter-5-scrubbing-data.html#bodies-and-headers-and-columns-oh-my . 
task_boiler.txt · Last modified: 2024/01/23 22:55 by raju