Artix Linux Forum

 Hello, I need to count some chars from a text file. Any wonder command to get this kind of output or I have to make a script?
 Thanks if anyone has a magic way it would be awesome! if can share it. Tried grep and some scripts but it gives poor results. Need this to make a script that belongs to a bigger project of mine about password quality strength. Thanks in advance  ;)

       
the input                                       the output
looks like                                      should look
   this                                                    this

  AAAAA                                               5
  AAAAB                                              4 1
  AAABB                                              3 2
  AAABC                                              3 1 1
  AAABD                                              3 1 1
  BAABE                                              2 2 1 

grep -oc "character" straight from the input?
Or as an alternative tr -dc "character" <<< input | wc -c
Iterate it for each line, then each character with a "while read -r" loop, then a "for x in a b c d" loop.

If you want to directly enumerate which charaters are on which line i'm afrain you will need awk.

 Cool, thanks, I'll check it out 👍🏻

 Found a script that worked for me
 

sed 's/[^A]//g' INPUT | awk '{ print length }' > OUTPUT

 Had to remove spaces and zeroes with tr

 And for the descending or ascending sort per each line tried sort command but it seems the way I did it didn't bring any mind blowing results  :D

sort -r -n FILE.txt

or

sort -r -g FILE.txt

for no avail
I'm sure there's a way of using sort in this case but I have no idea what flags I have to pass so to get desired result

Now I get like  all these 141 or 411 114 but I need descending like 411

iason:[nous]:~% cat t
AAAAA
AAAAB
AAABB
AAABC
AAABD
BAABE
iason:[nous]:~% perl -F'' -le '$c{$_}++ for @F; print %c; %c=();' t
A5
B1A4
A3B2
B1C1A3
A3D1B1
A2B2E1

#!/usr/bin/env python3
if __name__=='__main__':
	import sys
	with open(sys.argv[1],'r') as _:
		lines = _.readlines()

	for line in lines:
		#remove lead/trail whitespace; split on whitespace and rejoin without
		line = "".join(line.strip().split())
		counts = {}
		for c in line:
			counts.setdefault(c,0)
			counts[c] += 1
		print(' '.join((str(_) for _ in reversed(sorted(counts.values())))))

I used this on your sample and it ran like this

$ python312 counter.py counter.data
5
4 1
3 2
3 1 1
3 1 1
2 2 1

% perl -F'' -le '$c{$_}++ for @F; print %c; %c=();' t | sed 's/[^0-9]/ /g'
 5
 4 1
 3 2
 3 1 1
 1 3 1
 2 2 1

Not right for large numbers eg

$ cat t && perl -F'' -le '$c{$_}++ for @F; print %c; %c=();' t | sed 's/[^0-9]/ /g'
AAAAAAAAAAAAAAAAAAAA
AAAAB
AABBBBBBBBBBBBBBBBBBABB
AAABAAAAAAAAAAAAAAC
AAABBBBBBBBBBBBBBBBBBBD
BAABE
 20
 4 1
 3 20
 1 17 1
 1 3 19
 2 2 1

The python version gives this

$ python counter.py t
20
4 1
20 3
17 1 1
19 3 1
2 2 1

  Aaa thank you very much  :) . I'll look thru all of them tnx. In the meantime found a python script from this website (https://rosettacode.org/wiki/Permutations_with_repetitions)

 That generates arrangements with repetitions so virtually I can take that output, process it thru my requirements, for example print only strings with double aa in them.

 Script looks like this, it may be useful for those who wanna generate rep. arrangements.Now I'll look how I can use your solutions. Thanks again 👍🏻

'''Permutations of n elements drawn from k values'''

from itertools import product


# replicateM :: Applicative m => Int -> m a -> m [a]
def replicateM(n):
    '''A functor collecting values accumulated by
       n repetitions of m. (List instance only here).
    '''
    def rep(m):
        def go(x):
            return [[]] if 1 > x else (
                liftA2List(lambda a, b: [a] + b)(m)(go(x - 1))
            )
        return go(n)
    return lambda m: rep(m)


# TEST ----------------------------------------------------
# main :: IO ()
def main():
    '''Permutations of two elements, drawn from three values'''
    print(
        fTable(main.__doc__ + ':\n')(repr)(showList)(

            replicateM(7)

        )(['abcdefg'])
    )


# GENERIC FUNCTIONS ---------------------------------------

# liftA2List :: (a -> b -> c) -> [a] -> [b] -> [c]
def liftA2List(f):
    '''The binary operator f lifted to a function over two
       lists. f applied to each pair of arguments in the
       cartesian product of xs and ys.
    '''
    return lambda xs: lambda ys: [
        f(*xy) for xy in product(xs, ys)
    ]


# DISPLAY -------------------------------------------------

# fTable :: String -> (a -> String) ->
#                     (b -> String) -> (a -> b) -> [a] -> String
def fTable(s):
    '''Heading -> x display function -> fx display function ->
                     f -> xs -> tabular string.
    '''
    def go(xShow, fxShow, f, xs):
        ys = [xShow(x) for x in xs]
        w = max(map(len, ys))
        return s + '\n' + '\n'.join(map(
            lambda x, y: y.rjust(w, ' ') + ' -> ' + fxShow(f(x)),
            xs, ys
        ))
    return lambda xShow: lambda fxShow: lambda f: lambda xs: go(
        xShow, fxShow, f, xs
    )


# showList :: [a] -> String
def showList(xs):
    '''Stringification of a list.'''
    return '[' + ','.join(
        showList(x) if isinstance(x, list) else repr(x) for x in xs
    ) + ']'


# MAIN ---
if __name__ == '__main__':
    main()

That seems to be the golden answer. Tnx  ✅  ;)

 Tried  @replabrobin script but it shoots out only 2 columns, also didn't fully get why you use python312 or that is just a typo?

Maybe because you use different shell than bash or what can be the cause? You copied the code wrong I guess, idk  🤔

It is right, just unsorted, which is easily fixable. This version also gets rid of sed(1):

hyperion:[nous]:/tmp% cat t tt
AAAAA
AAAAB
AAABB
AAABC
AAABD
BAABE
AAAAAAAAAAAAAAAAAAAA
AAAAB
AABBBBBBBBBBBBBBBBBBABB
AAABAAAAAAAAAAAAAAC
AAABBBBBBBBBBBBBBBBBBBD
BAABE
hyperion:[nous]:/tmp% perl -F'' -le '$,=" "; $c{$_}++ for @F; print sort{$b<=>$a} values %c; %c=();' t tt
5
4 1
3 2
3 1 1
3 1 1
2 2 1
20
4 1
20 3
17 1 1
19 3 1
2 2 1

 
 EDIT: It worked @replabrobin  script I feed the wrong numeric format while should have used letters so that's why it gave me errors

 Now my issue is that I have a 24MB text file but it's a long uninterrupted string 7x800k letters . What I need now is to break this string starting from the beginning into just 7 chars long string from beginning to end. So I should have a column instead of that huge row, a column with 7 elements on each row. After that  I can feed that format to the count.py script

You're wasting people's time.

Mmmm NOPE, deduced something very important/crucial for my project. Solved that string breaking problem with fold -b7 etc so no problem with that.

The revealing conclusion is that calculation of a string pattern does not scale as I've expected. It's either impossible to deduce or just super hard to deduce/demonstrate mathematically. Without crash testing different strings lengths it's impossible to have an accurate picture of how/why/what changed the result.

I'll keep updating conclusions to my main thread on this topic for now. Tnx for patience and remember guys  nothing is in vane even if it may look and feel like that.  :(

 
 @nous I tried your perl command and it worked perfectly as a Swiss knife . I just tried @replabrobin cos python looks/sounds more friendly for me atm.

The only thing is when I issue your perl command like bellow

perl -F'' -le '$c{$_}++ for @F; print %c; %c=();' t | sed 's/[^0-9]/ /g' INPUT.txt > OUTPUT.txt

the shell throws 'no such file..' but   it shoots the right counts. So that error think it's because I don't perl it properly but  for reasons i don't quite get them rn 🤷🏻‍♂️

Anyway your perl command got saved  for later use in my  read only bash history file which means iz very gut 😋 ✅

There's a limit to how well an interpreted script can scale; after a certain point you need parallelization.
A quick'n'dirty Perl one-liner I tested on a 7-letter x 8 million times long string, 1 line file (which is 54MB) completes in 22 seconds. That's with splitting every 7nth character and counting occurrences, which is the computationally-expensive task. The Python script takes a little longer (29 seconds) on a pre-split version of the same file. A rewrite of the Perl version with fork() or MCE gives almost linear performance increase, in my 4c/8t CPU about 7 times faster and I imagine the same would apply to Python.

Your INPUT.txt is redundant (and missing, hence the error), the example reads from t; try the sorted version instead which as a bonus doesn't need sed:

perl -F'' -le '$,=" "; $c{$_}++ for @F; print sort{$b<=>$a} values %c; %c=();' INPUT.txt >OUTPUT.txt

  Aw thank you very much @nous, that one gonna land on my bash_history as well. Tnx for making it more clear to me  🤩

 Speaking about perl. Found this script which looks pretty slick. I'm trying to optimizing it for example to make it output multiple text files ,not bigger than 5MB instead of a one huge 1 GB text file or more. I need to test a 9x9 string which would require to generate 9^9 variants. Also would be cool if I can make this perl  script take some breaks so to give more air bubbles to the cpu so it will not start catch fire  lol. Script looks like this.

use Algorithm::Combinatorics qw/tuples_with_repetition/;
print join(" ", map { "[@$_]" } tuples_with_repetition([qw/A B C/],2)), "\n";

Now the output of this gives square brackets like  [A B] and in rows. Is there  a way to get rid of those brackets + shoot them directly in a column format? Tnx and sorry if I'm abusing of you guys precious time 😬

EDIT: looks like I need perl-math-combinatorics  module, omg, didn't see that coming..It's only on the AUR

Perhaps?

map { [@$_] }

No idea without the full script and the combi module. The square brackets are used to create an array reference, perhaps the double quotes retain the brackets too?

 Is there any easier way to write this formula in python or perl or whatever is more easy?

 Any of those three formulas basically are the same. Looks differently but does the same thing but I have no idea how they should look in python and I don't want to ask chatgpt either  :D

Tried to find a tool that converts mathematical formula to python but it seems there is no such a thing  ::)

The formula looks like in the image bellow. Any help is greatly appreciated 👍🏻 ☮

(https://i.postimg.cc/VLRWvzNT/gyfuyfuyoi8.png)

In pure Perl:

% perl -le 'sub fact{ $f=shift; $x=1; $x *= $_ for $x++..$f; return $x; } $T=13; $S=7; print (fact($T)/fact($T-($S-1)))*$S*($S-1)/2;'
1235520

Not even sure if it's correct. Also, doesn't check T and S for sanity.

 Thank you very much I'll check it and see how it works @nous is #1 🤩

 It seems that your perl script works but it solves only first part of the formula A[T,(S-1)] but has to multiply that with the second one

  S!/2(S-2)!  Thanks it seems it works but like I said would be nice if join into the perl party 🕺🏻 the second term 💃🏽 also

The one-liner follows the 3rd formula and it does calculate S(S-1)/2.

 Idk but the result gives only 13!/7! which is 1235520 that's the first formula factor A(13,6)

 The second is   6*7/2=21   

 The final result should be 1235520 × 21 = 25.945.920

 Your script must be doing gut only the first part but silently somehow discards the second, perhaps giving it the value of 1 instead of
 what it should be 21

I've quadrupled-triple checked swear on red 🫡

You're right, for some reason it stops before the second part, I put an extra pair of parentheses to help it:

perl -le 'sub fact{ $f=shift; $x=1; $x *= $_ for $x++..$f; return $x; } $T=13; $S=7; print ((fact($T)/fact($T-($S-1)))*$S*($S-1)/2);'

 
 Bingo, now it works as intended 👍🏻. Thank you @nous the King of Perl ✅. I'm gonna be asking for another implementation but first have to sort out what this script needs so we can just feed it with a T an S and the twos amount.

Another way is just sampling three of those a b c compare results and see if we need to scale up or down our twos amount from the string

Thanks once again, this is very helpful for my project 👍🏻 🤩

I'm only a novice, you'd be surprised if you saw what a real Perl monk can do with this swiss army chainsaw...

 
Then it means you are a novice reaching monk level pretty soon. Just look what beauty 'I have in store' for bumping up your Perl skills 😛

Tried to perl this myself but, hey I'm not even a perl novice 😄

So let's perl this I'll add a print screen. Of course take your time I know this must be challenging. If you can't  or don't have the time I'll let it here so hopefully other guys can give it a shot. There is no hurry, and of course with many thanks and gratitude 🤩 in advance

(https://i.postimg.cc/8PMdCYnX/vchcjkhg.png)

What the perl script steps has to do to solve the above formula is in the following order:

 1 Prompt user to input T  (what are the total elements, can be 94 or whatever or even bitcoin word list 2048)
 2 Prompt user to input S  (desired pwd length has to be   smaller than T so you can't feed a T=10 and an S=15)
 3 Prompt user input n (the number of twos he needs to check if is better or worse to have more or less twos in his pwd for the provided T and S))
 4 Determine Z with Z=(S-n) (both S and n are provided by the user in the above steps)
 5 Determine k with K=2*n   (k is twice as bigger than n)

 6 Proceed solve for  Λ that is the whole formula I'll reproduce it bellow one more time
 
      Λ = T!/(T-Z)!*S!/[(2^n)*n!*(S-k)!]

Added an example for verifying  the script if gives the proper result. Took an example like bellow:

when T=13 and S=7 and n=3 from user input we will have a Z=4 and  k=6

The result should be 17.160 * 105 = 1.801.800

#!/usr/bin/perl
use strict;
no strict 'refs';
use warnings;
use 5.36.0;

die "Usage: $0 <T: total elements> <S: pass length> <n: number of twos>" if $#ARGV < 2;

our ($T, $S, $n) = (int $ARGV[0], int $ARGV[1], int $ARGV[2]);

for (qw/T S n/) {
    die "$$_ not a positive integer" if !($$_ > 0);
}
die "Not T>S" if ($S > $T);
die "Not n<S" if ($n > $S);
die "Not S-n<T" if ($S-$n > $T);
die "Not 2n<S" if (2*$n > $S);

my $Z=$S-$n;
my $k=2*$n;
#        T! /    (T-Z)! *     S! / [(2^n) *   n!   *  (S-k)! ]
say fact($T)/fact($T-$Z)*fact($S)/((2**$n)*fact($n)*fact($S-$k));

sub fact {
    my $f = shift;
    my $x = 1;
    $x *= $_ foreach ($x++ .. $f);
    return $x;
}

% ./wtf.pl
Usage: ./wtf.pl <T: total elements> <S: pass length> <n: number of twos> at ./p.pl line 7.
% ./wtf.pl 13 7 3
900900

It gives half of what you said, one of us is mistaken somewhere.

 
Hmmm let me check it out, 1 minute please.

 Redone the calculation and gives me the same result 1.801.800. Now i'm looking into you script perhaps it can have some division by 2 somewhere

My best bet I can think of when looking at the script is that instead of dividing by 2^3 which is 8 it divides (2^3)*2=16  so I guess that's why gives half the result cos divides by 16 instead of just 8

 Here could  be the error (2**$n), just a guess though

The calc is this   13!/9!   ×  7!/2^3*6  = 17160 × 105 = 1.801.800

Also all number a integer natural number. There's no rational number anywhere in the calculation.

Saw the second term from the script gives 52.5  instead of 105 cos divides 105 by 2 so that's why result is half what it should

EDIT 3: Quadrupled-triple checked everything should work fine, no error on my end, its perl fault. Perl secretly hates math 😋

The error was in the fact sub, duh.

#!/usr/bin/perl
use strict;
no strict 'refs';
use warnings;
use 5.36.0;

die "Usage: $0 <T: total elements> <S: pass length> <n: number of twos>" if $#ARGV < 2;

our ($T, $S, $n) = (int $ARGV[0], int $ARGV[1], int $ARGV[2]);

for (qw/T S n/) {
    die "$$_ not a positive integer" if !($$_ > 0);
}
die "Not T>S" if ($S > $T);
die "Not n<S" if ($n > $S);
die "Not S-n<T" if ($S-$n > $T);
die "Not 2n<S" if (2*$n > $S);

my $Z=$S-$n;
my $k=2*$n;
#        T! /    (T-Z)! *     S! / [(2^n) *   n!   *  (S-k)! ]
say fact($T)/fact($T-$Z)*fact($S)/((2**$n)*fact($n)*fact($S-$k));

sub fact {
    return 0 if (my $f = shift) == 0;
    my ($i, $p) = (1, 1);
    $p *= $_ foreach ($i++ .. $f);
    return $p;
}

From now on you're on your own.  ;)

 Wow now works 1000%, amazing!!! I'll  retract, perl likes math 😀 👍🏻 but you have to be a super-blackhat-hacker. It works the only thing it doesn't do is the moment you input n=0 meaning no rep at all. But I'm ultra-gut with this 5 out of 5⭐ solution 💯. It's more than I could think of, wow 🍾 ✅ 🤩 🤩 🤩

I'll figure out that n=0

Many thanks @nous  #1 🌞 ☮

 
 Haha, made it in python. It may look dumb (i'm n00b) but works even when n=0. Also works when use T = 2048 bitcoin word list. Saw the perl script throws NaN in that case

# Online Python - IDE, Editor, Compiler, Interpreter
T = int(input("T: "))
S = int(input("S: "))
n = int(input("n: "))
Z = int(S)-int(n)
k = 2*int(n)
import math
a = math.factorial(T)
b = math.factorial(T-Z)
c = math.factorial(S)
d = math.factorial(n)*2**n*math.factorial(S-k)
e = int(a)*int(c)
f = int(b)*int(d)
g = (e/f)
print(int(g))

Hello it's me again  :D . I need to put some pauses when the above python script runs, I'm trying a 10x10 string which may drain too much resources  if run it without pause (10^10x10 bytes strings) Found bellow script that looks like it's good so I can ctr+c and pause the script then run it again pressing enter. Looks like it has a problem
1 idk where/how to insert it in my main py
2 the pause script looks like has some error print (...)

Also would b cool if somehow I can make the script dump results on disk and not keep it in memory like it seems to be doing till the end

Bellow the pause script found on stack (https://stackoverflow.com/questions/7180914/pause-resume-a-python-script-in-middle)

import time

while True:
    try:
        time.sleep(30)  # do something here
        print '.',

    except KeyboardInterrupt:
        print '\nPausing...  (Hit ENTER to continue, type quit to exit.)'
        try:
            response = raw_input()
            if response == 'quit':
                break
            print 'Resuming...'
        except KeyboardInterrupt:
            print 'Resuming...'
            continue

Thank you !!! 😇  No hurry 😇