Python: thread.error: can't start new thread

-
So the case is this: i have a python script that listens on certain port for a connection, and when a client connects, it acts as a tcp forwarder, forwarding connections accordingly. somehow, the server.py script throws exception after few client.py connection:


Unhandled exception in thread started by 

Traceback (most recent call last):

  File "server.py", line 55, in server_thread

    thread.start_new_thread(forward_client, (client_socket, server_socket))   

thread.error: can't start new thread


Damn it! so i do what all people been doing, i Google.

From below two URL, i was able to understand the situation and came up with the solution:

- http://blog.tsunanet.net/2010/10/threaderror-cant-start-new-thread.html
- http://adywicaksono.wordpress.com/2007/07/10/i-can-not-create-more-than-255-threads-on-linux-what-is-the-solutions/

Tsunanet states that memory allocation might be the issue. Sort of like where my RAM is not enough for all the threading happening. So acting l33t i follow what he (she?) did, strace:

mmap(NULL, 10489856, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)


Indeed! some memory allocation sort of thingy. but... what the hell is that? Wel.. /me dunno. There came Ady Wicaksono. From his blog, somehow i manage to understand that the formula to calculate the amount of threads that yr app can catter at any one time, is as below:

(total RAM size) / (ulimit -s size, which is the stack allocation, i think) = (number of thread)

Based on that, my RAM is 256MB, and current ulimit -s shows 10240 (10MB)
By calculation, i can only handle 25 threads! Damn it! no wonder python throws all those crappy errors..

So what i did? Simple: ulimit -s 1024 and my python app can ahdle 256 thread at one time. ok enough for me. :)

Ouh, as Tsunanet stae it, to make this permanent, ad below line in your /etc/security/limits.conf
(oh, worth to mention, im using Centos 5.5, 64-bit)

# added for more thread handling

# ref: http://blog.tsunanet.net/2010/10/threaderror-cant-start-new-thread.html



*       -       stack   1024  # 1024kB a.k.a. 1MB

Ok, hole this helps someone. Oh, and i'm on a VPS. :)

/alak

Comments

Post a Comment

Popular posts from this blog

SAP: Segregate Different SMTP Mail Server Based on Sender Domain

-
Image
  Requirement : An SAP server, with a single Productive client shared by multiple company codes. Each CoCode would have their own domain (FQDN), and any email send out, for example PO to vendors, are to be sent using each individual CoCode SMTP mail server. Solution Steps: 1. Configure multiple SMTP nodes in Tx SCOT, limiting each node with a Sender Group 2. Set different Sender Group (CDG) parameters for each user of each CoCode (Tx SU01 > Parameters) 3. Test using SCOT's Routing Test feature 4. Profit Detail Step: Logon to SAP system and configure multiple SMTP Node, can use  Continue with SCOT Wizard. Once you get multiple Nodes (as per below screenshot), double click any of them,  In this example, we double click on the "SECOND" node. And then click on "Set" button for the address type you want to configure (that would most probably be for INT type) And click the Pencil icon for setting the Sender Group Here we can set based on FQDN or any string. We are ...
Read more

SAP Client Copy Error: Errors occurred during export of container object FINB-TR-DERIVATION

-
 Reference from: https://www.consultoria-sap.com/2010/03/client-copy-derivation-rule-tables-are.html TL;DR - Execute report ABADRCHECK with selected 'DELSTEPS' //alak ==== Client copy is completed with status "Post-Processing Required". There is an error in log for object FINB-TR-DERIVATION: DA300 "No active nametab exists for *some db table*" Name of db table is usually with naming convention: xxyyyysssmmmnnnn (xx - application class, yyyy - strategy ID, sss - system indicator, mmm - client, nnnn - sequence number) These db tables are part of some instance of derivation tool in your system and are used to store Derivation Rule Values. Other terms ABADR, FINB_TR_CC_EXIT, FINB_TR_CC_EXIT_TARGET, DA 300, DA300, CC, SCC9, SCCL, SCC3 Reason and Prerequisites During processing CC there was huge workload into your system. CC processcreates new tables to store copied derivation rules. This is done in parallel asynchronous process. But due to workload into system, t...
Read more

Quick Dirty Script for Checking Oracle Data Gurad Gap and Emailing it

-
 Situation: primary DB: prd_CCC standby DB: prd_WWW file 1: gap-check.bat @echo off cls echo. echo. echo This script checks for Archive Logs at prd_ccc and Applied Logs at prd_www echo. echo Querying prd_ccc (Primary) echo select max(sequence#) "Primary" from v$archived_log where archived='YES';|sqlplus -s / as sysdba echo Querying prd_www (Standby) echo select max(sequence#) "Standby" from v$archived_log where applied='YES';|sqlplus -s / as sysdba echo. DGMGRL / "show configuration" date /t && time /t && hostname && whoami file 2: emailer.ps1 $Username      = "username" $EmailPassword = "password" $today = Get-Date $Username = $Username $EmailTo = "adam@jasabyte.com"  $EmailFrom = "server@jasabyte.com" $Body = Get-Content .\status.log -Raw $Subject = "PRIMARY/STANDBY DB GAP - $today" $SMTPServer = "mail.smtp2go.com"  $SMTPMessage = New-Object System.Net.Mai...
Read more