Skip to main content ↑ ↓
Topic solved
This topic has been marked as solved and requires no further attention.
Topic: Sed tr awk substitution problem (Read 2551 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

Sed tr awk substitution problem

 Hello guys despite having a new account i'm much older here on Artix forum. My ex lost account is Surf3r. I thought it might be helpful if i mention this as a side note. And now i want to jump into my question if anyone faced this kind of 'challenge'.
Long story short i need to substitute some numbers from a string i.e: [10 3 1 5 20 11] into some chars i.e:[ *~/!@+] So i thought tr can do the trick but i was wrong. Tried tr, sed, even some perl commands something like this

Code: [Select]
 echo "10 3 1 5 20 11' | tr '10' * | tr '3' ~ | tr '1' / | tr '5' ! | tr '20' @ | tr '11' +

However this not solve the problem as from what i saw sed/tr/ reads only one digit and not two like 10 11 20 and i get errors as i need each number (1-92) have its corresponding  char. I'm exactly trying to make a password generator  from 1-92 numbers and at the end desired string made out of numbers 1-92 (but only 63 long) to be translated into chars (symbols, a-z, A-Z, 0-9) that also count 92 elements.

Is there any way of solving this with sed or tr or awk or i have to look into making a full blown program doing this? I've searched the web tried finding something similar which looks pretty basic but for my surprise nobody had this kind of 'problem'

Re: Sed tr awk substitution problem

Reply #1 –
I think sed is more fit for this task. Also, the substitution order is important (i.e., if you first substitute 1 instead of 10, then you won't have and 10s left).
Code: [Select]
echo "10 3 1 5 20 11" | sed 's/10/*/g;s/3/~/g;s|1|/|;s/5/!/g;s/20/@/g;s/11/+/g'

[EDIT] Send me a message if you'd like to switch to your @Surf3r account.

Re: Sed tr awk substitution problem

Reply #2 –
Thank  you very much @nous for the solution that worked like a charm hope to see this applied to the hole 92 chars and see if works. I'll gonna contact ASAP via message and see if you guys can glue back my old account  :D . Tnx again and have a great day  8)

Re: Sed tr awk substitution problem

Reply #3 –
Another solution I can think of is using awk, which doesn't require any specific order.

Code: [Select]
{
  for (i=1;i<=NF;i++) {
    if ($i == 10) print "*"
    else if ($i == 3) print "~"
    else if ($i == 1) print "/"
    # etc.
  }
}

But why not get the 63 random characters for your password straight from /dev/random? This yields a far shorter command line using just tr:

Code: [Select]
tr -Cd 'a-zA-Z0-9[*~/!@+]' < /dev/random | head -c63; echo

Re: Sed tr awk substitution problem

Reply #4 –
Tnx for the second solution @capezotte  but it worked pretty smooth with sed. String generated the way you suggested  (100% random) is more likely 3-10 times weaker and can be even worse than that compared to a string generated following  my method that rules out completely any bad luck in getting a random low quality string .To summarize my method ensures a good quality string with no char repetition, all chars are different and ensures some proportions to blend your string behind the biggest number of possible arrangements. Have a look on this print screen i put together.

Those numbers    22|19|19| 3|   gives the biggest amount of arrangements for a 63 chars long pass

For example tested a GRC random generated password against my model and mine was almost 5.4 times better with significantly more arrangements than GRC's
(the only thing i forgot to add a char ^ to the recipe, to be fixed)

The only missing thing was that translation from numeric to chars.

Re: Sed tr awk substitution problem

Reply #5 –
It's worth studying how John the Ripper works to understand strong passwords. If you create any pattern you can write something to break that more easily, or just brute force random things anyway. For instance, your rules would make it possible to crack more easily, if it is known you say always used max or certain length passwords, then why waste time checking shorter ones first? If you never repeat characters then OK, don't waste time on any combinations that do repeat them. Any rule narrows down the possibilities. If you always mix numbers, special characters and letters, then it rules out all possibilities that don't include them. As for all that use 1 instead of i etc or add some numbers at the start or end of a word, well that's a real loser as it's so common even a dictionary attack can include those common tricks. Adjacent keys on the keyboard is another known pattern. There are lists of real passwords from leaked hacks online too, huge files of them to feed into your attack, a lot of the easily remembered things nobody would ever think of have been thought of. A lot of the strong password advice given by self proclaimed experts is rubbish, better than using really simple stuff but still not very good. Random is best but even knowing someone uses that makes it easier! Of course, in reality you also have to fit in with the demands of the interface you are writing a password for, as they often have particular requirements on length and content.

Re: Sed tr awk substitution problem

Reply #6 –
and then of course, if a password is so complex no one can remember it easily, it usually gets written down on a piece of paper at said person's workstation ie, a sticky note, so they can get in their own system. which represents one of the biggest security risks.  And if you work in a cubicle with an aisle behind you can likely never know who is standing behind you or not with absolute certainty, when you type it in..
Cat Herders of Linux

Re: Sed tr awk substitution problem

Reply #7 –
Quote from: ####### – on
It's worth studying how John the Ripper works to understand strong passwords. If you create any pattern you can write something to break that more easily, or just brute force random things anyway. For instance, your rules would make it possible to crack more easily, if it is known you say always used max or certain length passwords, then why waste time checking shorter ones first? If you never repeat characters then OK, don't waste time on any combinations that do repeat them. Any rule narrows down the possibilities. If you always mix numbers, special characters and letters, then it rules out all possibilities that don't include them. As for all that use 1 instead of i etc or add some numbers at the start or end of a word, well that's a real loser as it's so common even a dictionary attack can include those common tricks. Adjacent keys on the keyboard is another known pattern. There are lists of real passwords from leaked hacks online too, huge files of them to feed into your attack, a lot of the easily remembered things nobody would ever think of have been thought of. A lot of the strong password advice given by self proclaimed experts is rubbish, better than using really simple stuff but still not very good. Random is best but even knowing someone uses that makes it easier! Of course, in reality you also have to fit in with the demands of the interface you are writing a password for, as they often have particular requirements on length and content.
this is how the turing machine broke hitler's enigma machine, all correspondence ended with hail hitler.
Cat Herders of Linux

Re: Sed tr awk substitution problem

Reply #8 –
Quote from: ####### – on
It's worth studying how John the Ripper works to understand strong passwords. If you create any pattern you can write something to break that more easily, or just brute force random things anyway. For instance, your rules would make it possible to crack more easily..

I'll challenge any super hacker known to 'Universe' break a password with a chassis   |22|19|19| 3
Even by looking at the 'chassis' used they will simply not bother to break it. Sorry to discourage the guys with the 100% random pass kinda thinking, however that is not enough. first you have to choose the best back bone pattern sort to speak and second come call the RANDOM cavalry.
Generated a password with the help of GRC and it gave a pattern 22|20|15| 6 which  has more than 5x less arrangements meaning if my |22|19|19| 3| can be braked in 5 years GRC can be cracked in 1 year (assuming that can be possible)
According to my calculations not even 10k cluster of supercomputers can't break this kind of string.
Regarding repetition of a char even it gives more arrangements precisely arrangements with repetition and at first it might look like  better option is terrible wrong.
Take this example what you'll choose to simplify  a bit from 5 chars 1 2 3 4 5? You can use for your pass only 3 digits 1-5  i.e: 4 5 2 or
4 4 2? You can easily see even there are more arrangements with repetition for the 4 4 2 however it uses only 2 elements 4 and 2 while 4 5 2 imply a 3rd element which can't be braked if used only those 2 elements 4 and 2. In other words a string with repetition is significantly weaker even if it has bigger number of arrangements but having less elements is automatically lower quality.
If you choose more or less than 22 chars randomly from those 30 symbols automatically arrangements will go down 10x 100x or even more.
What i wanna point out we should not rely on randomness only but let's call it targeted randomness instead like a recipe where you add the exact amount of ingredients otherwise taste will not be that good. And why not having a better password if there is a way to make it unfeasible to break.
Of course if the hashing algorithm is not at the same level with the string quality of course it will be degrade the whole thing. From what i know for this kinda password it should be better if hashed with sha-512 as sha-256 (2^256) address space is a bit lower than total arrangements of a 63 char length pass A(92,63)

Re: Sed tr awk substitution problem

Reply #9 –
Tried with sed but no luck. Thought it would work but sed gets stuck and when i use [ ! ] symbol it throws an error and if i replace [ ! ] with [ ^ ] it does not produce any output. I checked with the letters and it seem to work, though sed does not like those symbols or whatever have no idea what  ???  This is exactly what i tried took me ages to write it. There are no typos but still sed gets stuck. Can't believe this substitution kinda basic op fails miserably and i don't get it why?

Code: [Select]
echo "14 92 57 19 56 87 55 27 36 69 35 20 73 41 18 24 52 70 62 5 65 54 23 16 77 60 72 50 64 13 71 38 59 79 80 78 28 37 47 40 21 25 17 88 1 51 29 63 2 6 76 15 3 4 33 42 68 9 66 46 39 48 32" | sed 's/92/9/g;s/91/8/g;s/90/7/g;s/89/6/g;s/88/5/g;s/87/4/g;s/86/3/g;s/85/2/g;s/84/1/g;s/83/0/g;s/82/Z/g;s/81/Y/g;s/80/X/g;s/79/W/g;s/78/V/g;s/77/U/g;s/76/T/g;s/75/S/g;s/74/R/g;s/73/Q/g;s/72/P/g;s/71/O/g;s/70/N/g;s/69/M/g;s/68/L/g;s/67/K/g;s/66/J/g;s/65/I/g;s/64/H/g;s/63/G/g;s/62/F/g;s/61/E/g;s/60/D/g;s/59/C/g;s/58/B/g;s/57/A/g;s/56/z/g;s/55/y/g;s/54/x/g;s/53/w/g;s/52/v/g;s/51/u/g;s/50/t/g;s/49/s/g;s/48/r/g;s/47/q/g;s/46/p/g;s/45/o/g;s/44/n/g;s/43/m/g;s/42/l/g;s/41/k/g;s/40/j/g;s/39/i/g;s/38/h/g;s/37/g/g;s/36/f/g;s/35/e/g;s/34/d/g;s/33/c/g;s/32/b/g;s/31/a/g;s/30/`/g;s/29/|/g;s/28/=/g;s/27/+/g;s/26/'/g;s/25/"/g;s/24/$/g;s/23/#/g;s/22/@/g;s/21/,/g;s/20/~/g;s/19/-/g;s/18/*/g;s|17|/|g;s/16/?/g;s/15/{/g;s/14/>/g;s/13/}/g;s/12/)/g;s/11/&/g;s/10/_/g;s/9/]/g;s/8/%/g;s/7/;/g;s/6/[/g;s/5/</g;s/4/(/g;s/3/^/g;s/2/./g;s/1/:/g'

Re: Sed tr awk substitution problem

Reply #10 –
Your sed expression is enclosed between ' and '. You try to substitute ' in the middle of that, so you're breaking the enclosure. If you remove that part (s/26/'/g), it works fine. I could provide you with the solution(s), but where's the fun in that?
Now that you know the cause, you can easily find the solution.

[EDIT]: typo

Re: Sed tr awk substitution problem

Reply #11 –
Ahaha thanks i got it i'll search for the problem think [ ' ] should be wrote like | ' | . I guess  :-[

After saw my incompetence in sed i found @capezotte  solution rather more elegant. something like this

Code: [Select]
tr -Cd '[*~/!@+]' < /dev/random | head -c22; echo

Code: [Select]
tr -Cd 'a-z' < /dev/random | head -c19; echo

Code: [Select]
tr -Cd 'A-Z' < /dev/random | head -c19; echo
   and last but not least

Code: [Select]
tr -Cd '0-9' < /dev/random | head -c3; echo

Re: Sed tr awk substitution problem

Reply #12 –
Unfortunately that way elements repeat themselves. (back on my old account, thank again @nous )


EDIT: found the problem in order to not get sed confused  instead of a single quote ' had to use [ '  "  '  "  ' ]  wrote with spaces between to be more readable. Long story short it worked. Tnx everybody for the helping ideas and solutions!!!

Code: [Select]
echo "14 92 57 19 56 87 55 26 36 69 35 20 73 41 18 24 52 70 62 5 65 54 23 16 77 60 72 50 64 13 71 38 59 79 80 78 28 37 47 40 21 25 17 88 1 51 29 63 2 6 76 15 3 4 33 42 68 9 66 46 39 48 32" | sed 's/92/9/g;s/91/8/g;s/90/7/g;s/89/6/g;s/88/5/g;s/87/4/g;s/86/3/g;s/85/2/g;s/84/1/g;s/83/0/g;s/82/Z/g;s/81/Y/g;s/80/X/g;s/79/W/g;s/78/V/g;s/77/U/g;s/76/T/g;s/75/S/g;s/74/R/g;s/73/Q/g;s/72/P/g;s/71/O/g;s/70/N/g;s/69/M/g;s/68/L/g;s/67/K/g;s/66/J/g;s/65/I/g;s/64/H/g;s/63/G/g;s/62/F/g;s/61/E/g;s/60/D/g;s/59/C/g;s/58/B/g;s/57/A/g;s/56/z/g;s/55/y/g;s/54/x/g;s/53/w/g;s/52/v/g;s/51/u/g;s/50/t/g;s/49/s/g;s/48/r/g;s/47/q/g;s/46/p/g;s/45/o/g;s/44/n/g;s/43/m/g;s/42/l/g;s/41/k/g;s/40/j/g;s/39/i/g;s/38/h/g;s/37/g/g;s/36/f/g;s/35/e/g;s/34/d/g;s/33/c/g;s/32/b/g;s/31/a/g;s/30/`/g;s/29/|/g;s/28/=/g;s/27/+/g;s/26/'"'"'/g;s/25/"/g;s/24/$/g;s/23/#/g;s/22/@/g;s/21/,/g;s/20/~/g;s/19/-/g;s/18/*/g;s|17|/|g;s/16/?/g ;s/15/{/g;s/14/>/g;s/13/}/g;s/12/)/g;s/11/&/g;s/10/_/g;s/9/]/g;s/8/%/g;s/7/;/g;s/6/[/g;s/5/</g;s/4/(/g;s/3/!/g;s/2/./g;s/1/:/g'

EDIT: still have problems with 1 and 11 it reads 1 as : but 11 it gives  : :  so still sed i don't see how it can be of any use cos that can happen to 2 and 22, 3 and 33 and so on  ::)

So 0-9 have to write them differently but still have no idea how

Re: Sed tr awk substitution problem

Reply #13 –
EDIT3: Found better method. Created 4 text files, one containing symbols second az, third AZ, forth 0-9. Used shuf

Code: [Select]
shuf symb -n 22 > test1   #text file called symb

Code: [Select]
shuf az -n 19 > test2   #text file named az
              
Code: [Select]
shuf AZ -n 19 > test3   # text file named AZ
               
Code: [Select]
shuf 09 -n 3 > test4   #text file called 09
                   
then used cat to merge outputs test1 test2 test3 test4 into a semifinal sfinal text file.
Code: [Select]
cat test1 test2 test3 test4 > sfinal
  and then  used shuf again
Code: [Select]
shuf sfinal > final   #this step can be repeated to mix the final string at the user discretion

the only thing now i have 'my precious' string  in column format and have no idea how to have it in a row format. :o  again everybody on the internet needs on the contrary in column format  ???

P.S. Wrote all steps in case there are people out there wanting to use my method without studding 100 years of intensive programming for a basic operation.

Re: Sed tr awk substitution problem

Reply #14 –
Quote from: Surf3r – on
EDIT: still have problems with 1 and 11 it reads 1 as : but 11 it gives  : :  so still sed i don't see how it can be of any use cos that can happen to 2 and 22, 3 and 33 and so on  ::)
So 0-9 have to write them differently but still have no idea how
Tip: s/11/&/ needs to be s/11/\&/ - some special characters need escaping.