investigate moving from CGI to django

one of the performance bottlenecks on the web interface is that we're using the old CGI interface which is unsustainable for 2 reasons:

it is a script based interface, so each time a script is called it spawns a new python interpreter (which means programs cant really be dynamic)

it is slow

instead, we should consider a modern web framework like django

The performance issue of OSNAP is not because of CGI. I tried to run the code below in python script (no CGI) in Tobler using the command like: python osnap_example_noCGI.py  and measured the duration.  And I found that It took 20 minutes to get the response. Please note that the code below does not contain any CGI, so it has nothing to do with CGI
============================================================
#!/home/suhan/public_html/cgi-bin/anaconda3/envs/osnap/bin/python
-- coding: UTF-8 --
enable debugging
import numpy
import json
import os
os.environ['QUILT_PACKAGE_DIRS']= "/home/suhan/public_html/cgi-bin/osnap/quilt_packages"
import osnap
import pandas as pd
Albertville = osnap.data.Community(source="ltdb",cbsafips="10700")
AlbertvilleDataframe = Albertville.census
out_data = Albertville.census[Albertville.census.year==1990]['p_hispanic_persons']
print(out_data)
**  Also, when I ran the code below (now it is CGI) on the Web, it took 20 minutes, too.**  This time I ran it like this : http://osnap.cloud/~suhan/osnap/python/osnap_example.py
===================================================================
#!/home/suhan/public_html/cgi-bin/anaconda3/envs/osnap/bin/python
-- coding: UTF-8 --
enable debugging
import cgitb
cgitb.enable()
import numpy
import json
import os
os.environ['QUILT_PACKAGE_DIRS']= "/home/suhan/public_html/cgi-bin/osnap/quilt_packages"
import osnap
import pandas as pd
print("Content-type: text/html\n\n")
Albertville = osnap.data.Community(source="ltdb",cbsafips="10700")
AlbertvilleDataframe = Albertville.census
out_data = Albertville.census[Albertville.census.year==1990]['p_hispanic_persons']
print(out_data)
On the other hand, my serverside program which includes CGI takes less than 1 minute.
Try here: http://sarasen.asuscomm.com/LNE/python/getLNAnalysis.py?&year=1990&state=01 IL&metro=10700&codes=1-03 hispXX
My serverside program which uses CGI does not slow down when the data are queried from the web. So the performance issue of OSNAP is not because of CGI. I guess the delay comes from the panda dataframe (not sure).
I know that Django is faster than CGI and it is newer than CGI. But the difference between two should be like 1 second per 1 query. 1 second performance improvement per every query might be a big deal. But I don't think that changing CGI to Django is the top priority that we need to do now to improve the performance of OSNAP because the problem does not come from CGI.

If I run things locally, I'm seeing like 15 seconds (not minutes):

But let's touch base on the issue during today's scrum.

Sorry. I was trying to write 20 seconds, not minutes.  Yes it took 20 seconds. 20 Seconds is a long time for users to wait to see a small choropleth map. Also, when we visualize the map, the visualization time will be added. My version (not osnap) with CGI takes less than 1 second. Let's talk about this issue, today.

my primary motivation for this comment was actually the first bullet

it is a script based interface, so each time a script is called it spawns a new python interpreter (which means programs cant really be dynamic)

with the CGI interface, each interaction with osnap is written as an individual script and executed as a its own python process. So there's no persisting the user state or holding data in memory for the user to explore with different visualizations etc.
We might be able to get around some of that by writing out temp files and such, but that has its own performance cost, and why reinvent the wheel? So it probably makes sense to move to a more dynamic framework anyway, and hopefully we should get a small speed boost for free

the performance difference between osnap and the custom implementation comes from the fact that osnap has to convert wkb into a geodataframe each time a new Community is instantiated--which is expensive. The custom webapp has geojson sitting ready in a database so it just needs to run a quick filter.
There are some straightforward ways we can speed up that process so that osnap will run faster in the web. But this is likely to always be a problem with the CGI interface, because you can't persist the geometries in memory. So you need that expensive conversion each time you make a CGI call

I agree that we are better to use Django for the things we develope for the future.  At this point, OSNAP (in python script) should run locally at a reasonable speed. No matter it is the first time or not, every query should run less than or around 1 second. Once we achieve it, I can move on to Django or whatever we want. Developing Django version with current OSNAP does not make sense to me because we won't be able to use it.  I am not sure if the architecture of OSNAP can finally beat the speed of the relational database.
Another solution is to make a batch program (in python) that brings data from OSNAP and automatically creates a relational database having the current database schema. But it is different from our original plan.

Another solution is to make a batch program (in python) that brings data from OSNAP and automatically creates a relational database having the current database schema. But it is different from our original plan.

this is essentially what is happening now. When you call osnap with CGI, it reads in a parquet file of all the tracts in the US and converts them into a GeoDataFrame, then runs a query to filter the data you selected. The queries execute in well under a second if the geoms are already in memory, and again, that's something we can do easily. But creating the spatial database and loading the geometries into memory takes a few seconds. With the CGI structure, we will never get around that bottleneck, because it needs to happen with each call.

Maybe I didn't get your point. I feel like we are talking about different things. But the program that I ran has nothing to do with CGI. I did not even import CGI. I ran the code below locally and took 20 seconds.  My understanding is that regardless of CGI or Django, it takes 20 seconds for this small query.
============================================================
#!/home/suhan/public_html/cgi-bin/anaconda3/envs/osnap/bin/python
#-- coding: UTF-8 --
#enable debugging
import numpy
import json
import os
os.environ['QUILT_PACKAGE_DIRS']= "/home/suhan/public_html/cgi-bin/osnap/quilt_packages"
import osnap
import pandas as pd
Albertville = osnap.data.Community(source="ltdb",cbsafips="10700")
AlbertvilleDataframe = Albertville.census
out_data = Albertville.census[Albertville.census.year==1990]['p_hispanic_persons']
print(out_data)
========================================================

right now you're calling a single python script, and the processing time is happening mostly in this line
Albertville = osnap.data.Community(source="ltdb",cbsafips="10700")

which does two things:

read the spatial data from a file and convert it to a dataframe -- this part takes time)
query and join two dataframes to select the data for Albertville (fips 10700) --this part is fast

we can make the instantiation of Community faster by speeding up step one. The way we speed that up is to load all the geometries into memory when you first import osnap. Then, each time a user instantiates a Community, they're really only executing step 2, which takes less than a second.
But you will always need to perform step 1 at least once. So with CGI (or in your current implementation where you're running a single python script, which is the same thing) that speedup never materializes because there's no way to load the geometries ahead of the user querying the data

I see your point. Does it mean that the user has to wait about 20 seconds for the first time even if the user wants to see the small choropleth map?

to do anything related to spatial data (including plotting it) osnap will always need to load the data into memory once, which takes ~15 seconds. That could essentially be hidden from the user by ensuring that when they access the web platform, there's already a python kernel ready with the geoms in memory.
(it's also worth pointint out that a lot of this discussion becomes moot, in the case of allowing users to upload/use their own data...)

I think the reason that I got confused was because I got the impression that Eli was saying that the performance problem comes from CGI. Now I clearly see that the problem in terms of the performance rises when I run the code below in Python Script. (Again it has nothing to do with CGI). We know that OSNAP data is cached after the first run in Jupyter, but it is not the case when we run locally OSNAP in Python Script. (Please see the code below). Please let me know if it is possible.
So I have two concerns now.

I think that there will be users who want to run OSNAP after they write the code in the python script in their desktop. In this case, the user will have to wait more than 20 seconds for every query since the data is not cached at the first time.

If the Web-developers must use Django in order to use OSNAP (since OSNAP data must be cached for a reasonable performance), they will not be able to add OSNAP modules to the codes that are already written based on CGI platform. So OSNAP will not be able to be widely used by Web-developers.

Django might bring the effect that Jupyter provides. The data might be cached after the first run on the Django frame as Jupyter does. I think that it is worth trying Django if we have enough time.  My worry is that I am 50:50 on it, Jupyter can be possibly the only frame that OSNAP data can be cached.

In terms of solutions,  I was thinking that we can probably develop another version OSNAP that utilizes a relational database. One of motivation is that the current architecture of OSNAP is hard to beat the speed of the relational database. I mean the way to use OSNAP will be the same between the two versions, but OSNAP relational database version will query the data from the relational database. I was thinking the new library that we can use just like OSNAP by using the same command line such as osnap.data.Community(source="ltdb",cbsafips="10700"), but brings the data from the PostGIS
The downside of the OSNAP relational database version will be that users will have to install DB in their machine (both can be done in Desktop or server), But I think that this version can be widely used by Web-developers. I think that there will be some web developers who want to add OSNAP module in their system which is already written in the platform other than Django. On the other hand, the current version of OSNAP will be widely used by Jupyter Users without the database.
But the downside of these ideas is that when we make the change in the library of OSNAP, we need to the same thing both in two different versions, and it will require a lot of time to develope OSNAP relational database version.
==Code ( this the code is not in CGI bin )=======================================
import timeit
import numpy
import json
import os
os.environ['QUILT_PACKAGE_DIRS']= "/var/www/quilt_packages"
tl = timeit.default_timer()
import osnap
import pandas as pd
Albertville = osnap.data.Community(source="ltdb",cbsafips="10700")
AlbertvilleDataframe = Albertville.census
out_data = Albertville.census[Albertville.census.year==1990]['p_hispanic_persons']
t2 = timeit.default_timer()
print(out_data)
print(t2-tl)
==The result======================================================
(osnap) suhan@tobler:~/public_html/Ex$ python osnap_example_noCGI.py
/home/suhan/public_html/osnap/osnap/osnap/data/data.py:44: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
df['geometry'] = df.wkb.apply(lambda x: wkb.loads(x, hex=True))
/home/suhan/public_html/cgi-bin/anaconda3/envs/osnap/lib/python3.7/site-packages/pandas/core/frame.py:3697: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
errors=errors)
/home/suhan/public_html/osnap/osnap/osnap/data/data.py:41: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
df['geometry'] = df.wkt.apply(wkt.loads)
geoid
01095030100    1.0
01095030201    1.0
01095030202    1.0
01095030300    0.0
01095030401    0.0
01095030402    0.0
01095030500    0.0
01095030600    0.0
01095030701    0.0
01095030702    0.0
01095030801    1.0
01095030802    1.0
01095030902    1.0
01095030903    1.0
01095030904    1.0
01095031000    1.0
01095031100    0.0
01095031200    0.0
Name: p_hispanic_persons, dtype: float64
20.08569398149848

I think now I have about 4 concerns.

I see that the problem in terms of performance rises whenever I run OSNAP in Python script since the datasets are not cached for the first run.  Jupyter caches the dataset for the first run, but it is not the case when we run the same code in Python script. Django might provide the effect that Jupyter provides, which means the dataset might be cashed at the first run on Django framework. But I am not sure that Django is the solution. I am 50:50 on it. Jupyter possibly might be the only framework which allows data to be cached at the first run. Also, since I am new to Django, it will take some time to learn how to install and use Django in a way that we want. I currently have no idea about how we can make LTDB data in OSNAP can get cached at the first time query within the Django framework.

If users (mostly web developers) must use Django to use OSNAP, they will not be able to add OSNAP to the codes that are already written based on CGI framework. For example, I am one of the representatives of many users, I originally hoped to simply replace OSNAP in my current framework (which is all python script), but in reality, now I am in the situation that I need to replace the whole framework to Django (not sure if Django will be a solution, tho). It would be really nice for users like me to add OSNAP to my current framework without performance issues. In addition, even if somehow I manage to make the version that can be cached based on Django, it requires me to load OSNAP asynchronously at the beginning of the website loading, and it is not easy to achieve it. It will also require many changes on both clientside and serverside code.  If I do not make this change, the user will experience 20 second waiting when they try to see the map for the first time on the Web.

I think that there will be many users ( among non-web developers) who want to add OSNAP in their python script and run their python script (not in Jupyter) in their desktop. In this case, the users will have to wait 20 seconds for every query.

My overall impression is that OSNAP is optimized for Jupyter users and it can be really easy to use on the Jupyter environment because users can use it without installing the database ( which is a big merit). But this framework is not so friendly for Python users who run their python codes in script either on desktop or servers because of the 20 minute waiting time for every query.

In terms of solutions, I was thinking that we might need to develop another version of OSNAP that reads the data from the relational database. I was thinking like another version of OSNAP that queries the same data by using the same command such as osnap.data.Community(source="ltdb",cbsafips="10700"), but read the data from PostGIS. I think that this OSNAP relational database version will be able to widely used by Web-developers and desktop users who want to run OSNAP in non-Jupyter environment.  But the downside of this idea is that it will require a lot of time to develope.

rdb vs pandas

More concerns:
If users (mostly web developers) must use Django to use OSNAP, they will not be able to add OSNAP to the codes that are already written based on CGI framework. For example, I am one of the representatives of many users, I originally hoped to simply replace OSNAP in my current framework (which is all python script), but in reality, now I am in the situation that I need to replace the whole framework to Django (not sure if Django will be a solution, tho). It would be really nice for users like me to add OSNAP to my current framework without performance issues. In addition, even if somehow I manage to make the version that can be cached based on Django, it requires me to load OSNAP asynchronously at the beginning of the website loading, and it is not easy to achieve it. It will also require many changes on both clientside and serverside code. If I do not make this change, the user will experience 20 second waiting whenever they try to see the map for the first time on the Web.
I think that there will be many users ( among non-web developers) who want to add OSNAP in their python script and run their python script (not in Jupyter) in their desktop. In this case, the users will have to wait 20 seconds for every query.
My overall impression is that OSNAP is optimized for Jupyter users and it can be really easy to use on the Jupyter environment because users can use it without installing the database ( which is a big merit). But this framework is not so friendly for Python users who run their python codes in script either on desktop or servers because of the 20 second waiting time for every query.

to clarify a couple points:

i dont mean to suggest that we need to go with django--I'm sure there's a newer sexier option. I just think it's important that we keep in mind that we will eventually need to move away from CGI. It's just not flexible enough for the dashboard-like interface we need (which is why dash is another important option to consider)--but it is useful for planning out the calls to osnap we need the web interface to make and the kind of structures osnap will return that need to be styled in the browser

it's important to be clear that the bottleneck in osnap is not from querying data (the link above shows pandas is actually considerably faster than postgres). The slowdown comes from reading data from flat files into memory. osnap is an analytics library, not a data repository, so a small slowdown when reading a huge data file into memory is acceptable--especially because we can optimize that part elsewhere in the library and because users working with other data will have a delay when they upload new data anyway (keep in mind we can't provide direct access to the LTDB data, even in a webapp). In short, there are several reasons I don't think we should be fixated on the 15 second load time while we are prototyping the web interface. I'm much more concerned about choosing an infrastructure that gives us the best layout & navigation tools, and has the interactivity that CGI lacks

also, web developers can consumer osnap however they like. Just because we're building a visualization front end using a particular library doesnt mean other developers couldn't build a CGI-based app using osnap

Eli. Thank you for your clarification.

I agree that we need to move away from CGI about the program that we will develop in the future.
I understand the performance bottle neck comes from reading data. Also, my understanding is that visualization of LTDB data in maps and charts on the web is fine unless we distribute the data out of our database. On the web app version, we do not need to ask users to upload LTDB data.
I still prefer to start working on the replacement of OSNAP modules once the performance issue is resolved because we do not know if Django or another frame should be used to tackle the performance bottleneck.

Eli and Serge
The core issue is that I do not know how to cache out LTDB data in OSNAP. The caching never happens when I run OSNAP in Python Script. Without resolving it, it will take at least 20 seconds for every query since it reads the whole LTDB data for every query request. It would be really nice if someone can come up with a good solution, but at this point, I do not know how long it will take for me to find out a good solution. Since I do not know the architecture of OSNAP, it is really hard for me to quickly come up with the solution.  Even though the performance of panda is generally better than Postgres, I do not see this benefit of Pandas in OSNAP because OSNAP currently reads the whole data for 20 seconds per every query in Python script.
If you guys think that I still need to work on replacing current modules querying LTDB data with the current modules of OSNAP without resolving the performance issue at this point, I can move on it as soon as I wrap up the client-side programming. (It will still be after a few weeks later from now because it is more efficient to complete what I am doing before I move on to the next thread).
But the programs that I can make at this point will take at least 20 seconds + visualization time for every query since the OSNAP will read the whole LTDB data for every query. ( Again, I currently don’t know how to cache out LTDB data at the beginning of the program). I, as a web developer, personally think that this is like making a garbage because I feel like it is like developing something that will never be used because of slowness. In this case, I personally think that it is better to invest my time to work on other threads instead of developing a server-side programs that is much slower than the current ones.  Because of this reason, I honestly do not have a good motivation to work on it as it is. Also, it is why I would like to start working on replacing the current modules with OSNAP once I see that the LTDB can be cached in the Python Script. But I think that you guys can have a different perspective from me, which can probably work out (hopefully).  So I am also okay to do whatever you would like to do.
I think that I have explained the current bottleneck of OSNAP enough when it runs in Python script and what's going to happen after we replace the current server-side program with OSNAP: it will take at least 15 seconds per every query unless we come up with a good solution to cache out LTDB data at the beginning.  Even though it happens, I can still add OSNAP in my current serverside program if it is something that you would like to do.  it can be done around March deadline. Moving from CGI to Django will add another time. (Also, we don’t know if Django can be a solution)
Otherwise, I can explore some of solutions that enable caching out LTDB data, but in this case, I do not know if I can come up with a good solution. I might not be able to come up with the solution even after a few weeks of exploration. Anyway, the decision is upto you.

I think the root of the problem is there's no way to cache the data when working with osnap using the CGI interface. When you use CGI, each python script executes independently, so there's no way to cache the geometries in memory so that they are available to different python processes. One thing we can do to improve this is provide functions that allow osnap to pull data from different backends. Right now, when you instantiate a Community, it automatically pulls tract geometries from a parquet file, which is fast enough for desktop analysis but leaves room for improvement in the browser. We could, for example, add an argument to Community that accepts postgres credentials, which will grab geoms from a database to try and speed up the data ingestion/conversion.
A better way to improve this pain point, I would argue, is to use a different web framework that keeps a python process alive across user sessions (rather than spawning a new process each time a user executes a script). We could still provide functionality to pull geoms from a database, but if we arent running each python process independently, it will greatly improve the application's performance (and generally make things easier on our end, because we can persist objects in memory to re-query, or build additional plots, etc)
But the important point, I think, is that we just shouldn't be too bothered about the speed at this stage. There are several opportunities to improve performance, but right now we can make the most progress by understanding how a user will interact with osnap on the web (i.e. what the UI needs to look like, and what it needs to accomplish) and whether we want to support certain kinds of workflows (like critical plotting functions) that arent yet implemented. What matters most right now is getting the infrastructure in place and understanding how osnap hooks into it. Once we have that in place, we can work on optimizing the connections between osnap and the web

My finding is that LTDB data is never be cached when we run OSNAP in python script regardless of desktop or Web. Even on the desktop, OSNAP in python script does not cache LTDB data. Jupyter is the only platform where LTDB data can be cached.

In the current implementation the data are never cached, even in jupyter. Instead, each time a user isntantiates a Community, the raw data are read in from a flat file.
This is an intentional design decision that increases processing time but decreases memory overhead. It is not required, and there are many ways we could cache the shape data in memory outside of jupyter. We can go over this more in the meeting today


Enable GitHub actions to publish to GitHub Pages

Looks like the workflow failed to publish to GitHub Pages: https://github.com/emscripten-forge/sample-python-repl/actions/runs/4795486188/jobs/8530117589
@DerThorsten I don't have the rights on this repo to enable the GitHub Pages so would you mind doing it?
You can find it under the Settings. Here is an example with another repo:

Thanks!

done! Thanks!

Cool! Now it should be possible to trigger a new build.


HCI socket ERROR

Hi,
I'm currently doing a project on BLE mesh networks, and your interface seems perfect for this.
However I keep getting the following errors:
Initialising...
Bluez down failed
Bind failed
HCI socket ERROR - no Bluetooth?
What would the problem be and how can I fix it?

Yes, this should have been in the documentation - btferret must be run with root privileges from root or via sudo as follows:
sudo ./btferret

btferret must be run with root privileges from root, or via sudo as follows:
sudo ./btferret
Thanks for flagging up this issue, the documentation will be updated.


Optimize extract_clusters and prepare_clusters systems

Objective
When developing my game I realized extract_clusters and prepare_clusters systems are taking a lot of time despite me creating very little lights. Reducing number of clusters from the default 4096 to 2048 or less greatly improved performance and stabilized FPS (~300 -> 1000+). I debugged it and found out that the main reason for this is cloning VisiblePointLights in extract_clusters system. It contains light entities grouped by clusters that they affect. The problem is that we clone 4096 (assuming the default clusters configuration) vectors every frame. If many of them happen to be non-empty it starts to be a bottleneck because there is a lot of heap allocation. It wouldn't be a problem if we reused those vectors in following frames but we don't.
Solution
Avoid cloning multiple vectors and instead build a single vector containing data for all clusters.
I've recorded a trace in 3d_scene example with disabled v-sync before and after the change.
Mean FPS went from 424 to 990. Mean time for extract_clusters system was reduced from 210 us to 24 us and prepare_clusters from 189 us to 87 us.

Changelog

Improved performance of extract_clusters and prepare_clusters systems for scenes where lights affect a big part of it.

Can you run this on some of our more intensive stress test examples? In this particular case, many_lights might be a better test than 3d_scene.

Here you are - tracy comparison for many_lights:

I showed data for 3d_scene first (which may be not something that we want to optimize) because it has small amount of lights (just 1) that has entire scene in range. As I understand it it is a pessimistic case where nearly all clusters are in range of a point light causing a lot of Vec allocations.
In many_lights example we have a lot of very small lights (radius = 0.3) so most of the clusters are empty and in such case cloning vectors is not a bottleneck.
I am not sure how much this optimization would affect real projects. I think it may affect some with indoor areas or if someone tries to model sun as a point light. But even if it was for simple examples like 3d_scene I think it is worth merging because of the impact on FPS. Low FPS (when compared to other engines) in simple scenes may discourage people from using Bevy.

CI failure appears to be a network error; retrying the merge.


Switching order of -a and -b paramter in bedtools intersect when -b file is BAM file, outputs BAM header.

cd test/intersect

bedtools  intersect -a j1.bed -b oneUnmapped.bam
chr1	98650	98704	a1	100	+

$ bedtools  intersect -b oneUnmapped.bam -a j1.bed
chr1	98650	98704	a1	100	+
�BC�Y]lW>�I���"�V/(�����k��]�v�fm�MM-���z��gg�8!UBE J�R)R�~���	�
                                                                                *���T�?��*/�P��ϝg��;~8��w���~{Ϲ���7'n|
                                                                                `������,��Jkm��y�nߵ�Neym��\gz���Jku��=3���	Ape�5��âݳǻ�-��z^m���k��aH" �BP�ji��BD���E@�9#���!�Bsb��h�@�%�aj����a�$�
�̠����	��L4��H�9e&��$�1!�c��$BR�cKpd�,��i� #� �2��
                                                         p�D8�T�
	$���:4�!�,�'X+֐@��L�0�����%��tB%!
���kw�=�޶�p�Ĉ>��f�b   �e���!�I/F���&[a!Dh�v�;z !{8��qC�v�D�JNLĵ���Qn�=��`3\������ ��E��n�F�3�b��B2^$
                                                    �!��)8�:W����
                                                                 �4Z#�uF{h�n��E,J�j�t��
�<��QJ��a�/o�o@q�����o7�sX\��*�#l�7�N-���x���S'f։�M��L('VƉ@�*�Dd�TG���BGRk��U�K���֫DXy�^]
�.��ޱa!�=q��3�M�%{yY������ꛌ[���r�aI�Cyɒ_��(���C�2�^��_�1Z��(��,��#����î�3���R|�[��uN7�GM���B��Zej����Q�l�tn���d�ֹ�R��1+���&
                                                                                ��{(�G�1S`��"�G-�\_��K�q��RQ*~���йR<&R|�?��-w|��E�Jp���G�[L�F���~B�R��!�5D�ç�DɤV���F�52x)�Ө�悕
                                                            >j������x��rO��Z�������[ZMu
iG�(W4h�:J-R*x���K��E�:YzJцE����[=b��X�X�yR*�,b�l7ʭnD ����δƕ)<f�Y��`�,��Tʳ˕���Յ���7�
                                                                                ��ҁw��jh�����7���s+��a�L*Tg����7t���2?�w��{3��o�#�[�9���z_}��]b���h�3^�5߬��Q_�}q\��ZC����������+n��u�<8� -��wŝ����
���>���+n�z�����#�W����\��8C�n�ذ���`{��n�+N�A"����ڨW������䧵�VϬ-��9��7�.y���c�}���5T�x��������χ��;����dp��[��>0�ت:�x��]M�4P
�*���,=�Ү�셲��#����`����~��9\�����~�]����bK@ Rc3b��)q����%8&���4�x2<q�KJ�����|_,0���U�����t,��I�z
                                                                                _�Ă?=
                                                                                ��b���xQ�>��D��*�O$wv�~��(l\��q%1����D�0���ɀ��lS�	o����Q�s�n$��g�{|*���L$
              �ޟ'�D��w'��-<��js���;_U�&�:�����cd��݉�Q�X��
                                                          '�q�ӡ/\��(Ln�
                                                                        �nM��j��ߚ����$�W��W�R����[�WR������K*�������~�&�p�Zƅ�S�7���L��
<7S��7�]�F�0����o%��hn�'rF5������˘N������P�Q�{p7��GQ1��Ϟ�7A��\��S��&�\c"oM����&��$}���z�M���{Rkn<ܙ�[�
���ɼ��__����cj���@�a��j��@m����_�/P                                                                           �&���9pj��)
                                      x�ռZ^ Ӈj
                                              ��j�
                                                   vI(���5��?/Ps�eM��\��盘p����ڂ���@-��ͼ����
                                                                                ���)P��sj
�?.P3h}����
            ���jN��`�������j����
                                     Ë?,P�*�	��|�@��_�
�N?Y�6`���	K�(P[�S�[�ַ�
                            d���"�BCu

It might be a more general problem with the parsing order of the arguments.
Example 1 should give the same result as example 4.
Example 2 should give the same result as example 3.
$ printf 'chr1\t123456\n'
chr1	123456

$ printf 'chr1\t123456\t123457\n'
chr1	123456	123457

# Example 1
$ bin/bedtools intersect -a <(printf 'chr1\t123456\n') -b <(printf 'chr1\t123456\t123457\n')
ERROR: file /dev/fd/63 has non positional records, which are only valid for the groupBy tool.

Tool:    bedtools intersect (aka intersectBed)
Version: v2.26.0-50-gccdc919-dirty
Summary: Report overlaps between two feature files.

Usage:   bedtools intersect [OPTIONS] -a <bed/gff/vcf/bam> -b <bed/gff/vcf/bam>
...

# Example 2
$ bin/bedtools intersect -b <(printf 'chr1\t123456\n') -a <(printf 'chr1\t123456\t123457\n')
ERROR: file /dev/fd/63 has non positional records, which are only valid for the groupBy tool.

Tool:    bedtools intersect (aka intersectBed)
Version: v2.26.0-50-gccdc919-dirty
Summary: Report overlaps between two feature files.

Usage:   bedtools intersect [OPTIONS] -a <bed/gff/vcf/bam> -b <bed/gff/vcf/bam>
...

# Example 3
$ bin/bedtools intersect -a <(printf 'chr1\t123456\t123457\n') -b <(printf 'chr1\t123456\n')
ERROR: Received illegal bin number<PHONE_NUMBER> from getBin call.
ERROR: Unable to add record to tree.

# Example 4
$ bin/bedtools intersect -b <(printf 'chr1\t123456\t123457\n') -a <(printf 'chr1\t123456\n')
Segmentation fault (core dumped)

After the patch for the first issue, the output of example 4 changed.
# Example 4
$ bin/bedtools intersect -b <(printf 'chr1\t123456\t123457\n') -a <(printf 'chr1\t123456\n')
***** WARNING: File  has inconsistent naming convention for record:
chr1	123456

***** WARNING: File  has inconsistent naming convention for record:
chr1	123456

I encountered the same error and I found that my bed files were prepared with windows type "End Of Line".
Lesson learnt:
make sure the bed files are prepared with linux type end of line (EOL).
Solution: open the file in notepad++
How to check: View>show symbols>show end of line. CRLF -> windows and LF-> linux.
How to convert: Edit>EOLconversion>Unix(LF)

the error was
ERROR: file sorted_YH3K9me3.bed has non positional records, which are only valid for the groupBy tool.

Also see dos2unix on the cli.
I think this can be closed?


No pragma once

Done via:
git grep -l '^# *pragma  *once$' | xargs perl -pi -e 's/^# *pragma once\n//'

Checklist
Thank you for your contribution. Please answer the following:

[x] I acknowledge that I am authorized to submit this code under
the terms of the Apache License
[ ] All new public APIs are documented
[ ] All new public functions have tests
[ ] All new public functions have fuzzing tests

Added a fixup commit to undo the PEM file modificaton.

Had to rebase and fix conflicts, #170 removed some files that this PR editred.

You make one little git mistake ...  sigh. fixup pushed.


Add Sidekiq job ID and logging to data exports

Some data exports mysteriously failed on production. It's not clear whether this happened because the job wasn't enqueued properly, whether it silently died when it ran, or for some some other reason.
Couldn't reproduce locally, so will start adding instrumentation until we know more about what happened here. This PR:

adds a sidekiq job ID to the data export. This might help us track down the job if something goes wrong — or at least assure us that it was enqueued.
adds logging inside the Sidekiq worker so we can catch any silent failures

Context
https://ukgovernmentdfe.slack.com/archives/CN1MCQCHZ/p1605518196399900

Woopsie, I think this is leaking between sidekiq jobs:
https://kibana.logit.io/app/kibana#/doc/8ac115c0-aac1-11e8-88ea-0383c11b333a/logstash-2020.11.19?id=fhuA33UB2W7S_aTlHlYj&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-4h,to:now))
I think we need to clear the variables we set before each job or something like that?


Default values not respected

Seems like default values are not respected when I perform an insert like so:
userDao.insert(user).handleAsync { ... }

Strangely, if I use the SQL builder wrapped in a vertx.executeBlocking, default values are subsequently inserted properly:
val got = dsl.insertInto(USER, USER.EMAIL, USER.PASSWORD).values(email, password)

I assume that this is related to the setDefault method in AbstractVertxDAO, but I'm not entirely sure that's where the problem lies.

Postgres: 10.2
Vertx-jooq version: 3.0.1
Vertx-jooq flavour: completablefuture-async

Thanks for the report.
You are right it: should set default values when the column is not nullable
and is no identity column. Can you confirm this applies to the column you
want to be defaulted?
Am 05.03.2018 1:26 vorm. schrieb "Gurpreet Paul"<EMAIL_ADDRESS>
Seems like default values are not respected when I perform an insert like
so:
userDao.insert(user).handleAsync { ... }
Strangely, if I use the SQL builder wrapped in a vertx.executeBlocking,
default values are subsequently inserted properly:
val got = dsl.insertInto(USER, USER.EMAIL, USER.PASSWORD).values(email, password)
I assume that this is related to the setDefault method in AbstractVertxDAO
https://github.com/jklingsporn/vertx-jooq/blob/0d997a8ba54cb3be7b7871a2fa885f4ed2fc5bec/vertx-jooq-shared/src/main/java/io/github/jklingsporn/vertx/jooq/shared/internal/AbstractVertxDAO.java#L220,
but I'm not entirely sure that's where the problem lies.
Postgres: 10.2
Vertx-jooq version: 3.0.1
Vertx-jooq flavour: completablefuture-async
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jklingsporn/vertx-jooq/issues/39, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGC83naw2wxW4galKa8AN87xviVywrGvks5tbIZJgaJpZM4SblIl
.

You're right. I added NOT NULL to the Postgres column and it worked!
Thanks.

I see, many thanks.


Update news

The news section on our website has not been updated for some time. We need to refresh it with the latest updates, aligning it with our recent LinkedIn posts. Include, but not limited to, the following list:

[ ] Quantum Optimization Working Group Meeting at ZIB @pedromxavier
[ ] MIP workshop @AlbertLee125 @anurag-r20
[ ] ESCAPE / PSE @ZedongPeng
[x] Welcome new members @parkyr and @anurag-r20
[ ] 2024 INFORMS Optimization Society Conference

After discussion today at the group meeting, we decided to not include all the conference news and only have the major updates here.
Let's rather link to LinkedIn and clean it up a little bit


Added multi-database support

Based on #452 - with suggested changes.

@bartekrylko ENUM and SET have been added via #464. Please rebase and squash your branch (and force push it to update the PR), as it has gotten rather cluttered with merges.

Ehh, this is shitty commit. Don't look at it. Will prepare something better.

New PR was created for that feature #503


Hint toward provisioning token, proxy, etc. configs when deploying node sensor

It's a common error to not set additional falcon parameters that the CID or environment requires. In the docs when deploying the node sensor, make a note of those and direct to the FalconNodeSensor docs.

we need to modify the whole operator yaml to make it work with all these parameters

Hi @kaushik853 this issue is just to add some additional language in the deployments docs to guide users toward a few specific parameters in the FalconNodeSensor reference docs: https://github.com/CrowdStrike/falcon-operator/blob/main/docs/resources/node/README.md#falconnodesensor-reference-manual
Do you have additional feedback or a question on their use?

ohhk..i get that..so in my set up while deploying nodesensor, the provisioning token was not getting accepted...as the heredoc ">>EOT"...until i specified as !!str...or you create the heredoc as yamlencode


Historical data visualizations (Improvement)

It would be nice to store the availability each time vacinespotter checks for availability and then display the historical data. Useful information could be displayed such as the number of available appointments over time and/or correlated by the time of day. This can help users know the best time to check for new appointments (and also see how quickly appointments fill)
(I may be able to contribute to this effort, if the feature is approved)

+1 for this.  I've been watching the JSON files for each state to get some idea of what the changes look like but access to more complete data (I know it's a lot) would be very helpful for spotting trends in when these providers drop appointments.


script for loadStoryblokBridge is injected twice

Describe the issue you're facing
I was trying out the storyblok integration for astro (with v5 beta) as was noticing this warning in the console:

[!caution]
Uncaught SyntaxError: Identifier 'loadStoryblokBridge' has already been declared (at page.js:14:24)

During dev this not an issue, but when building an error gets also thrown:

[!caution]
astro:scripts/page.js (14:23): Identifier "loadStoryblokBridge" has already been declared
file: astro:scripts/page.js:14:23

12:
13:
14:               import { loadStoryblokBridge, syncContentUpdate } from "@storyblok/astro";
                          ^
15:               loadStoryblokBridge().then(() => {
16:                 const { StoryblokBridge } = window;

 Hint:
   Browser APIs are not available on the server.

   If the code is in a framework component, try to access these objects after rendering using lifecycle methods or use a `client:only` directive to make the component exclusively run on the client.

   See https://docs.astro.build/en/guides/troubleshooting/#document-or-window-is-not-defined for more information.

It seems like this is related to this section:
https://github.com/storyblok/storyblok-astro/blob/c555207a9b93989655431a95c57e5d7ab762f628/lib/index.ts#L193-L226
But looking at the config it seems fine and in line with the flow control here, so no idea why the script is injected both times, unless the resolved config is somehow different.
Reproduction
https://github.com/oddcelot/astro-storyblok
Steps to reproduce
No response
System Info
"@storyblok/astro": "5.1.0-alpha.4",
"astro": "5.0.0-beta.3"

Used Package Manager
pnpm
Error logs (Optional)
No response
Validations

[X] Follow our Code of Conduct

So i did try and patch this via pnpm patch and removing the additonal import seems to resolve the issues for dev and build 🎉
https://github.com/oddcelot/astro-storyblok/commit/73855c864b048120704a8c1da372213b90bacdb7

Hey @oddcelot, thank you very much for opening this issue and looking into it. @dipankarmaikap and I currently working on a new major release of @storyblok/astro and will address this.


Fix link prefix for homepage

If the user supplies a http/https prefix then the homepage link breaks because of it. Do not supply a prefix and let the user choose.

Thanks, but dupe of #126.

@OJFord do you know when it'll be merged (or if @posquit0 is maintaining this repo at all)? The PR seems to be stuck for the last 7 years...

Posquit0 added me to help with the workload, but unfortunately I don't spend a lot of time on it either - I've had a couple of big clear outs and tried to stay on top of new requests (especially just closing down / moving to discussions) the spurious or unrelated stuff, which frankly is the bulk of it, but I haven't been able to get through a lot of the backlog of genuinely useful work or suggestions.
Since it seems the author there has moved on, I will either merge as is or apply my suggestion myself. I'm just always cautious to test manually any changes myself since I don't cause inadvertently 'breaking changes' (in this context really any changes to style or layout with existing use of the class unchanged) in people's CVs.


incorrect dashboard header in toh documentation

toh-pt5.jade displays an image (/resources/images/devguide/toh/dashboard-top-heroes.png) that contains an incorrect header. The header reads 'My Dashboard', when it should read 'Top Heroes'

Can you take a peek @kapunahelewong ?

I've included this update with a PR fixing another issue in the same doc. Thank you very much @swseverance!!


Jhu raw publisher

Tested and worked
2020-05-28 12:25:16.731  INFO 22151 --- [           main] c.s.covid19.Covid19CasesRawPublisher     : Starting Covid19CasesRawPublisher on tamimialtamimi.local with PID 22151 (/Users/taltamimi/covid19-stream-processors/stream-processors/jhu/COVID19RawPublisher/target/classes started by taltamimi in /Users/taltamimi/covid19-stream-processors/stream-processors/jhu/COVID19RawPublisher)
2020-05-28 12:25:16.733  INFO 22151 --- [           main] c.s.covid19.Covid19CasesRawPublisher     : No active profile set, falling back to default profiles: default
2020-05-28 12:25:17.276  INFO 22151 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2020-05-28 12:25:17.279  INFO 22151 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
2020-05-28 12:25:17.282  INFO 22151 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2020-05-28 12:25:17.310  INFO 22151 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-05-28 12:25:17.316  INFO 22151 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-05-28 12:25:17.317  INFO 22151 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-05-28 12:25:17.526  INFO 22151 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8091 (http)
2020-05-28 12:25:17.536  INFO 22151 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-05-28 12:25:17.537  INFO 22151 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.30]
2020-05-28 12:25:17.610  INFO 22151 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-05-28 12:25:17.610  INFO 22151 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 846 ms
2020-05-28 12:25:17.802  INFO 22151 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-05-28 12:25:18.063  INFO 22151 --- [           main] c.f.c.c.BeanFactoryAwareFunctionRegistry : Looking up function 'publishRawData' with acceptedOutputTypes: []
2020-05-28 12:25:18.094  INFO 22151 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2020-05-28 12:25:18.107  INFO 22151 --- [           main] c.f.c.c.BeanFactoryAwareFunctionRegistry : Looking up function 'publishRawData' with acceptedOutputTypes: []
2020-05-28 12:25:18.110  INFO 22151 --- [           main] c.f.c.c.BeanFactoryAwareFunctionRegistry : Looking up function 'publishRawData' with acceptedOutputTypes: []
2020-05-28 12:25:18.111  INFO 22151 --- [           main] c.f.c.c.BeanFactoryAwareFunctionRegistry : Looking up function 'publishRawData' with acceptedOutputTypes: [application/json]
2020-05-28 12:25:18.198  INFO 22151 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2020-05-28 12:25:18.199  INFO 22151 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'application.errorChannel' has 1 subscriber(s).
2020-05-28 12:25:18.199  INFO 22151 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean '_org.springframework.integration.errorLogger'
2020-05-28 12:25:18.199  INFO 22151 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {router} as a subscriber to the 'publishRawData_integrationflow.channel#0' channel
2020-05-28 12:25:18.413  INFO 22151 --- [           main] o.s.integration.channel.DirectChannel    : Channel 'application.publishRawData_integrationflow.channel#0' has 1 subscriber(s).
2020-05-28 12:25:18.413  INFO 22151 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean 'publishRawData_integrationflow.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-05-28 12:25:18.549  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connecting to host 'orig=tcp://localhost:55555, scheme=tcp://, host=localhost, port=55555' (host 1 of 1, smfclient 2, attempt 1 of 1, this_host_attempt: 1 of 21)
2020-05-28 12:25:18.573  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connection attempt failed to host 'localhost' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - Error communicating with the router. cause: java.net.ConnectException: Connection refused ((Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - )
2020-05-28 12:25:21.575  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connecting to host 'orig=tcp://localhost:55555, scheme=tcp://, host=localhost, port=55555' (host 1 of 1, smfclient 2, attempt 1 of 1, this_host_attempt: 2 of 21)
2020-05-28 12:25:21.576  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connection attempt failed to host 'localhost' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - Error communicating with the router. cause: java.net.ConnectException: Connection refused ((Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - )
2020-05-28 12:25:24.580  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connecting to host 'orig=tcp://localhost:55555, scheme=tcp://, host=localhost, port=55555' (host 1 of 1, smfclient 2, attempt 1 of 1, this_host_attempt: 3 of 21)
2020-05-28 12:25:24.582  INFO 22151 --- [           main] c.s.j.protocol.impl.TcpClientChannel     : Connection attempt failed to host 'localhost' ConnectException com.solacesystems.jcsmp.JCSMPTransportException: (Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - Error communicating with the router. cause: java.net.ConnectException: Connection refused ((Client name: tamimialtamimi.local/22151/#001a0001/DWHwBSD_oV   Local port: -1   Remote addr: localhost:55555) - )


Fix fab dbshell

'fab server manage_run:dbshell' was not working. I guess there
wasn't enough information in the Django settings for Django to
invoke psql correctly to connect to our databases the way they
are set up in the PSF infrastructure.
Added a new fab task 'dbshell' that extracts the needed info
the same way we do before running a dump, and invokes psql
directly.  Also tweaked 'manage_run' so if 'dbshell' is passed
as the command, it uses the new task.

I get this error when running:
$ fab staging manage_run:dbshell
[pycon-staging.iad1.psf.io] Executing task 'manage_run'

Warning: Unable to load SSH config file '/home/lovewell/.ssh/config'

Traceback (most recent call last):
  File "/home/lovewell/.virtualenvs/pycon/local/lib/python2.7/site-packages/fabric/main.py", line 739, in main
    *args, **kwargs
  File "/home/lovewell/.virtualenvs/pycon/local/lib/python2.7/site-packages/fabric/tasks.py", line 317, in execute
    multiprocessing
  File "/home/lovewell/.virtualenvs/pycon/local/lib/python2.7/site-packages/fabric/tasks.py", line 213, in _execute
    return task.run(*args, **kwargs)
  File "/home/lovewell/.virtualenvs/pycon/local/lib/python2.7/site-packages/fabric/tasks.py", line 123, in run
    return self.wrapped(*args, **kwargs)
  File "/home/lovewell/projects/PyCon/pycon/fabfile.py", line 49, in manage_run
    exec(dbshell)
TypeError: exec: arg 1 must be a string, file, or code object

Any ideas?

On Mon, Aug 10, 2015 at 11:19 AM, Rebecca<EMAIL_ADDRESS>wrote:

File "/home/lovewell/projects/PyCon/pycon/fabfile.py", line 49, in manage_run
exec(dbshell)
TypeError: exec: arg 1 must be a string, file, or code object
Any ideas?

Fixed.

:game_die:


Fix dot as main

I'm going to close this for now. @airportyh, if you (or anyone else) can provide a failing test, I'll be more than happy to reopen this.


Prepend path separator to GOROOT prefix when checking for stdlib packages

previously, this code was simply checking if the build.Default.GOROOT was a prefix of the target package path to see if the package was from the stdlib. this was failing for me, as the default GOROOT /usr/local/go was being used (since I don't have one set), but all of my packages lived under /usr/local/google/....
sadly, there's no better solution in the filepath package to check if a path is nested under another path.

Could you add a test case for this please?

It sounds like this is deceptively hard to do right: https://github.com/golang/go/issues/18358
I suspect this change might cause a bug if the user does define $GOROOT and includes a trailing slash. Perhaps using filepath.Clean() on both paths would fix this? I'd like to see some test cases to get some confidence that this is an improvement.


It is impossible to get into devMode

 options.devMode = yargs.argv.dev || false;
this makes setting devMode to true from config file impossible.

Not yet. Eventually when in dev mode events will get logged so you can observe them.

This has been fixed and there is now events logged in dev mode on join and leave of channels.


[IRGen] Query the VFS for legacy type layouts.

All file I/O performed by the frontend should go through the VFS.

@swift-ci please test

@swift-ci please test

@swift-ci please test

@benlangmuir am I holding it right?


[feature] add relative time for weeks

Also rollup v0.68.0 breaks the build so avoided it in the package.json
check PR test for more details.
To replicate:

remove node_modules.
do npm install on develop branch.
watch test fail

Coverage decreased (-0.07%) to 88.109% when pulling 57a9383f2e1ace305daccc3754652bd5b1ad566d on nanomosfet:add-weeks-to-relative into b8f4fe2bf5bb9022655505d59b62aef12bc540d6 on moment:develop.

Coverage decreased (-0.07%) to 88.109% when pulling 57a9383f2e1ace305daccc3754652bd5b1ad566d on nanomosfet:add-weeks-to-relative into b8f4fe2bf5bb9022655505d59b62aef12bc540d6 on moment:develop.

Could this get a review again? @ichernev @marwahaha

Is this feature going to be merged sometime soon?

Merged in 9bd2dc74209ec21c327aa2af6620e3d2dd3ae4b6


Consider the restriction that relationships may only be between leaf nodes

There seems to be an implict restriction that relationships may only exist between leaf nodes types.
Consider:
Node type A has a child node type A'
Node type B has a child node type B'
Relationship X has a child relationship type X'
Node Type A is related via X to Node Type B
The topology comprises Node instances of Type A' and Type B'
I would expect node instance A' to be related to instance B' via relation instance of type X via inheritance but implementations I have experienced would not look for any relationships between A' and B' unless they are explictly defined in A' and/or B'

@pmjordan: Is the following template illustrated your issue?
tosca_definitions_version: tosca_simple_yaml_1_3
capability_types:
  C: {}
relationship_types:
  X:
    valid_target_types: [C]
  Xx:
    derived_from: X
node_types:
  A:
    requirements:
      - x:
          capability: C
  A':
    derived_from: A
  B:
    capabilities:
      c:
        type: C
  B':
    derived_from: B
topology_template:
  node_templates:
    n1:
      type: A'
    n2:
      type: B'

Parsing this file with Cloudnet TOSCA toolbox produces the following trace:
$ translate issue_94.yaml
Translate issue_94.yaml...
[Info] issue_94.yaml: TOSCA syntax checking
[Info] issue_94.yaml: No service template catalog to load.
[Info] issue_94.yaml: TOSCA type checking...
[Warning] issue_94.yaml@14,23:node_types:A:requirements:x:relationship: undefined but X or Xx are compatible with C!
[Info] issue_94.yaml: topology_template:node_templates:n1:requirements:x - n2 node template found
[Info] issue_94.yaml: TOSCA type checking done.
....

So the requirement x of n1 is fulfilled by n2 as shown in the following generated diagram:

So what is the issue?

Thank you, yes that is a good example. It seems that the restriction I refer to is only present in some implementations. So the question now is should the standard be explict that this inheritance is execpted behavior for all implemenations?

From what I understand the issue seems to be confined to one implementation - and has been resolved as a bug. So I'm closing this issue.


How to config and build game for iOS

Hi @gmhevinci
Could you please add some simple note on how to config and build game for iOS platform?
Thank you so muh!
I am trying to switch platform to iOS and press build on AssetBundle Builder, then export the demo, when run on xCode i have this error:
Failed to download web data : file:///private/var/containers/Bundle/Application/33E5C3AF-02E4-4E73-B5E4-E1CF8B2F8289/gametest.app/Data/Raw/PatchManifest.bytes Error : HTTP/1.1 404 Not Found
System.Action`2:Invoke(T1, T2)
MotionFramework.Network.d__1:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)
Exception: Fatal error : Failed download file : file:///private/var/containers/Bundle/Application/33E5C3AF-02E4-4E73-B5E4-E1CF8B2F8289/gametest.app/Data/Raw/PatchManifest.bytes
at MotionFramework.Patch.LocalBundleServices+d__4.MoveNext () [0x00000] in <00000000000000000000000000000000>:0
at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00000] in <00000000000000000000000000000000>:0
(Filename: currently not available on il2cpp Line: -1)

I just try again by pressing three buttons after changing the Platform to iOS and can build and run.
Thank so much!

If the PatchManifest.bytes is not found, please make sure it is included in the StreammingAssets folder.


Possible GPU Memory Leak - Mem Usage Keeps on Increasing with Every Iteration

I'm on Ubuntu 16.04, CUDA 9.1, CUDNN 7.1, one P6000 GPU (compute 6.1) with 16 Gb VRAM, Python 3.6, Pytorch 0.4 and all other required libraries are successfully installed and properly working.
Tried batch sizes 4, 8, 12, 16, 24, 44 and 48.
Always get out of memory error after some time.
Training starts, and GPU VRAM usage keeps on increasing with every iteration.
I tried correcting a couple of lines in layers.py and another line in train.py following what was said in deprecation warnings.
Still get out of memory errors.
Tried disabling CUDNN in hparams after reading something about memory leaks with Pytorch and CUDNN.
Still get out of memory errors.
I have a private dataset that mimicks the exact attributes, properties and structure of LJSpeech.
Tried using all 480K WAV train files, 20K WAV test files and 5K WAV validation files.
Also tried using 12.5K WAV training files, 1K WAV test files and 500 WAV validation files (same as LJSpeech).
Still get out of memory errors.
What could be the problem?
Would greatly appreciate some help regarding this.
Thanks in advance!

Thank you for reporting this.
Can you build a docker container using this docker file and let us know if the error persists?
Regarding running with CUDA 9.1 and CUDNN 7.1:
Are you using the default values in hparams?
Can you pin point where the increase in memory is occurring with the default hparams?
My two guesses are that it occurs at some += operation or that it occurs in some operation in the  LocationLayer in model.py.

@rafaelvalle
This happened again:
Train loss 254 0.974303 Grad Norm 1.096113 18.54s/it
THCudaCheck FAIL file=pytorch_1524584710464/work/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "train.py", line 291, in
args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 216, in train
y_pred = model(x)
File "torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs)
File "model.py", line 510, in forward encoder_outputs, targets, memory_lengths=input_lengths)
File "python3.6/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs)
File "model.py", line 403, in forward decoder_input)
File "model.py", line 363, in decode attention_weights_cat, self.mask)
File "python3.6/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs)
File "model.py", line 77, in forward attention_hidden_state, processed_memory, attention_weights_cat)
File "model.py", line 60, in get_alignment_energies processed_query + processed_attention_weights + processed_memory))
RuntimeError: cuda runtime error (2) : out of memory at pytorch_1524584710464/work/aten/src/THC/generic/THCStorage.cu:58 (AIWN9)
Will now try the updated dockerfile ...

@rafaelvalle
Still got the error even with the dockerfile. Here's what happened with batch_size 12:
Train loss 107 0.993192 Grad Norm 0.723081 11.45s/it
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu
line=58 error=2 : out of memory
Exception ignored in: <bound method _DataLoaderIter.del of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f4070db6da0>>
Traceback (most recent call last):
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 349, in del self._shutdown_workers()
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 328, in _shutdown_worker self.worker_result_queue.get()
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/queues.py", line 337, in get return _ForkingPickler.loads(res)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd fd = df.detach()
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach with _resource_sharer.get_connection(self._id) as conn:
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection c = Client(address, authkey=process.current_process().authkey)
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/connection.py", line 494, in Client deliver_challenge(c, authkey)
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/connection.py", line 722, in deliver_challenge response = connection.recv_bytes(256)        # reject large message
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/connection.py",line 216, in recv_bytes buf = self._recv_bytes(maxlength)
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4)
File "/home/MX9/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
File "train.py", line 291, in  args.warm_start, args.n_gpus, args.rank, args.group_name, hparams)
File "train.py", line 216, in train
y_pred = model(x)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input,**kwargs)
File "/opt/AITTSv3/nv-wavenet/pytorch/tacotron2/model.py", line 512, in forward mel_outputs_postnet = self.postnet(mel_outputs)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py",line 491, in call result = self.forward(*input, **kwargs)
File "/opt/AITTSv3/nv-wavenet/pytorch/tacotron2/model.py", line 144, in forward x = self.dropout(F.tanh(self.convolutionsi))
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 53,in forward return F.dropout(input, self.p, self.training, self.inplace)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py",line 552, in dropout return _functions.dropout.Dropout.apply(input, p, training, inplace)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 36, in forward ctx.noise = cls._make_noise(input)
File "/home/MX9/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 10, in make_noise return input.new().resize_as(input)
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58
Perhaps max length? Would greatly appreciate some help. :)

Still haven't found the cause of this.
I've tested two machines already.
Machine A has 1 P6000 16Gb VRAM.
Machine 2 has 1 Tesla V100 16Gb VRAM.
...
Have also placed torch.cuda.empty_cache() and gc.collect() in main loop.
Still get CUDA out of memory error.
Have tested downgrading to CUDA 8 and CUDNN 6, installed Pytorch 0.4.0 CUDA80 and Tensorflow 1.3 (and 1.6) for machine A.
CUDA out of memory error still persists.
...
Anybody encountering the same problem with a private dataset?

Finally! I've identified the cause of this problem.
Despite running my getlength.py script through my raw dataset to separate a dev dataset, it still got mixed up with some audio files with lengths beyond the largest lengths of the files in the LJSpeech-1.1 dataset.
Now running 64 batch size in machine with 1 Tesla V100 16Gb VRAM GPU.  :)
Closing this now ...
BTW, largest length of files in LJSpeech-1.1 is 10092 ...

Resolved issue by re-separating files from raw private dataset to dev dataset with max lengths the same as max lengths of files in LJSpeech-1.1 ...

@MXGray can you share your optimizations for handling CUDA memory and RAM?


+/- resource "aws_api_gateway_deployment" "apigateway-lambda_api-gateway-deployment_3AF9B512" {
      ~ created_date  = "2024-02-06T20:21:03Z" -> (known after apply)
      ~ execution_arn = "arn:aws:execute-api:us-east-1:996905175585:b1aoz7w8yj/" -> (known after apply)
      ~ id            = "ypr8t4" -> (known after apply)
      ~ invoke_url    = "https://b1aoz7w8yj.execute-api.us-east-1.amazonaws.com/" -> (known after apply)
      ~ triggers      = { # forces replacement
          ~ "deployedAt"   = "1707250813894" -> "1707252844926"
            # (1 unchanged element hidden)
        }
        # (1 unchanged attribute hidden)
    }

  # aws_api_gateway_stage.apigateway-lambda_api-gateway-stage_E95B0DF9 will be updated in-place
  ~ resource "aws_api_gateway_stage" "apigateway-lambda_api-gateway-stage_E95B0DF9" {
      ~ deployment_id         = "ypr8t4" -> (known after apply)
        id                    = "ags-b1aoz7w8yj-prod"
        tags                  = {}
        # (9 unchanged attributes hidden)
    }

  # aws_lambda_function.apigateway-lambda_fxa-events-apigateway-lambda_events-lambda_06645171 will be updated in-place
  ~ resource "aws_lambda_function" "apigateway-lambda_fxa-events-apigateway-lambda_events-lambda_06645171" {
        id                             = "FxAWebhookProxy-Prod-ApiGateway-FxA-Events-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:FxAWebhookProxy-Prod-ApiGateway-FxA-Events-Function:847" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:FxAWebhookProxy-Prod-ApiGateway-FxA-Events-Function:847/invocations" -> (known after apply)
        tags                           = {}
      ~ version                        = "847" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"            = (sensitive value)
                # (3 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

  # aws_lambda_function.proxy-lambda_fxa-events-sqs-lambda_39345609 will be updated in-place
  ~ resource "aws_lambda_function" "proxy-lambda_fxa-events-sqs-lambda_39345609" {
        id                             = "FxAWebhookProxy-Prod-Sqs-FxA-Events-Function"
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:996905175585:function:FxAWebhookProxy-Prod-Sqs-FxA-Events-Function:880" -> (known after apply)
      ~ qualified_invoke_arn           = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:996905175585:function:FxAWebhookProxy-Prod-Sqs-FxA-Events-Function:880/invocations" -> (known after apply)
        tags                           = {
            "environment" = "Prod"
            "service"     = "FxAWebhookProxy"
        }
      ~ version                        = "880" -> (known after apply)
        # (19 unchanged attributes hidden)

      ~ environment {
          ~ variables = {
              ~ "GIT_SHA"     = (sensitive value)
                # (4 unchanged elements hidden)
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 1 to add, 3 to change, 1 to destroy.
aws_lambda_function.apigateway-lambda_fxa-events-apigateway-lambda_events-lambda_06645171: Modifying... [id=FxAWebhookProxy-Prod-ApiGateway-FxA-Events-Function]
aws_lambda_function.proxy-lambda_fxa-events-sqs-lambda_39345609: Modifying... [id=FxAWebhookProxy-Prod-Sqs-FxA-Events-Function]
aws_lambda_function.apigateway-lambda_fxa-events-apigateway-lambda_events-lambda_06645171: Modifications complete after 6s [id=FxAWebhookProxy-Prod-ApiGateway-FxA-Events-Function]
aws_api_gateway_deployment.apigateway-lambda_api-gateway-deployment_3AF9B512: Creating...
aws_lambda_function.proxy-lambda_fxa-events-sqs-lambda_39345609: Modifications complete after 6s [id=FxAWebhookProxy-Prod-Sqs-FxA-Events-Function]
aws_api_gateway_deployment.apigateway-lambda_api-gateway-deployment_3AF9B512: Creation complete after 0s [id=ez9jft]
aws_api_gateway_stage.apigateway-lambda_api-gateway-stage_E95B0DF9: Modifying... [id=ags-b1aoz7w8yj-prod]
aws_api_gateway_stage.apigateway-lambda_api-gateway-stage_E95B0DF9: Modifications complete after 1s [id=ags-b1aoz7w8yj-prod]
aws_api_gateway_deployment.apigateway-lambda_api-gateway-deployment_3AF9B512 (deposed object bd05c182): Destroying... [id=ypr8t4]
aws_api_gateway_deployment.apigateway-lambda_api-gateway-deployment_3AF9B512: Destruction complete after 0s

Apply complete! Resources: 1 added, 3 changed, 1 destroyed.


Wheel support

I'm distributing my software using pynsist.
It allows to import packages on a chosen version only if a .whl is available on pypi.
It would be much more easy to distribute.
Would it be possible to do it and publish future releases with a wheel file ?

Posted release 0.6.6 with .whl file. Please check it on your side. Especially with Python 2.7.

Just tested with python 3.7.1 and it works perfectly.
Unofortunately I have not the possibility to try it with python 2.7.


multi: extract expiry and perms from macaroon

For the expiry to be properly extracted, Litd must be on https://github.com/lightninglabs/lightning-terminal/pull/378
the extraction of the permissions from the macaroon works but is a bit of a hack at the moment. Hoping to get this a bit cleaner

cc @guggero, keen to hear if you know how i can do the last commit in a less hacky way. The macaroon permissions are encoded in the macaroon ID in proto form. But this proto definition is in an internal directory in the macaroon package and so cant easily unmarshal the ID to get the []bakery.Op permissions out 🙈 so I had to hack out the macaroon permissions in a pretty ugly way.


Session establishment failed

Hi !
I just used the demo from the docs, compiled and failed:
Exception: Sharp.Xmpp.XmppErrorException: Session establishment failed.

client.Connect();

The xmpp server record is there and properly read. The credentials for
the jid are verified. My chatroom appender with agXMPP works properly
with that server, which is an OpenFire.
Thanks anyway and best regards,
Manfred

Hi,
It would be very useful if you could append a log from the server side. I could direct you to related steps (please note that these are different and depend on the OS version used for openfire)

Hello !
Thanks for your reply.
I additionally ask a question about OpenFire's log in the OpenFire forum, because it shows
something, which I never saw before.
OpenFire is running on Debian 6.0.10 (squeeze) with version 3.6.4
OpenFire has two log entries, under warning and under debug.
Warning:
2015.09.29 17:38:01 Closing session due to incorrect hostname in stream header. Host: thor.manne.eu. Connection: org.jivesoftware.openfire.net.SocketConnection@a946ac socket: Socket[addr=/<IP_ADDRESS>,port=57507,localport=5269] session: null
Debug:
2015.09.29 19:36:24 NIOConnection: startTLS: using c2s
2015.09.29 19:36:25 AuthorizationManager: Trying Default Policy.authorize(test<EMAIL_ADDRESS>2015.09.29 19:36:25 DefaultAuthorizationPolicy: Checking authenID realm
2015.09.29 19:36:25 DefaultAuthorizationPolicy: authenRealm = xmpp.domain
2015.09.29 19:36:25 LocalOutgoingServerSession: OS - Trying to connect to thor.manne.eu:5269(DNS lookup: thor.manne.eu:5269)
2015.09.29 19:36:25 LocalOutgoingServerSession: OS - Plain connection to thor.manne.eu:5269 successful
2015.09.29 19:36:25 Connect Socket[addr=/<IP_ADDRESS>,port=57607,localport=5269]
2015.09.29 19:36:25 Connection closed before session established
Socket[addr=/<IP_ADDRESS>,port=57607,localport=5269]
2015.09.29 19:36:25 LocalOutgoingServerSession: OS - Going to try connecting using server dialback with: thor.manne.eu
2015.09.29 19:36:25 ServerDialback: OS - Trying to connect to thor.manne.eu:5269(DNS lookup: thor.manne.eu:5269)
2015.09.29 19:36:25 Connect Socket[addr=/<IP_ADDRESS>,port=57608,localport=5269]
2015.09.29 19:36:25 ServerDialback: OS - Connection to thor.manne.eu:5269 successful
2015.09.29 19:36:25 ServerDialback: OS - Sent dialback key to host: thor.manne.eu id: bc6e5a19 from domain: manne.eu
2015.09.29 19:36:25 ServerDialback: RS - Received dialback key from host: manne.eu to: thor.manne.eu
2015.09.29 19:36:25 ServerDialback: RS - Error, hostname not recognized: thor.manne.eu
2015.09.29 19:36:25 Connection closed before session established
Socket[addr=/<IP_ADDRESS>,port=57608,localport=5269]
2015.09.29 19:36:25 ServerDialback: OS - Unexpected answer in validation from: thor.manne.eu id: bc6e5a19 for domain: manne.eu answer:<stream:error xmlns:stream="http://etherx.jabber.org/streams"></stream:error>
2015.09.29 19:36:25 OutgoingSessionPromise: Error sending packet to remote server:

java.lang.Exception: Failed to create connection to remote server
at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.sendPacket(OutgoingSessionPromise.java:252)
at org.jivesoftware.openfire.server.OutgoingSessionPromise$PacketsProcessor.run(OutgoingSessionPromise.java:216)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Also the connected was close - so it looks for me, by the client - the java-arch cannot cancel
the started thread at this point. As sad, I asked in there forum, but have no answer yet. As sad,
all my agXMPP base tools ar working from inside the firewall and also Gajim and PSI. From
outside, I use Xaber and Jitsi. My DNS record are ok too:
Do:
host -t SRV _xmpp-client._tcp.manne.eu
host -t SRV _xmpp-server._tcp.manne.eu
and this is the result:
_xmpp-client._tcp.manne.eu has SRV record 10 10 5222 thor.manne.eu.
_xmpp-server._tcp.manne.eu has SRV record 10 10 5269 thor.manne.eu.
The testprogram runs on Windows Server 2008 R2 with anti-Virus protection
and firewall enabled. I had disbaled both, just to be sure and added an additional
rule for the testprogram.
Hope, this gives a tip.
OOOHH, suddenly, at least, something comes to my mind: My server does not have
a official certificate !!!!!
Thanks so far and
best regards,
Manfred
From: PGS<EMAIL_ADDRESS>Sent: Tuesday, September 29, 2015 4:29 PM
To: pgstath/Sharp.Xmpp
Cc: mabra
Subject: Re: [Sharp.Xmpp] Session establishment failed (#9)
Hi,
It would be very useful if you could append a log from the server side. I could direct you to related steps (please note that these are different and depend on the OS version used for openfire)
—
Reply to this email directly or view it on GitHub https://github.com/pgstath/Sharp.Xmpp/issues/9#issuecomment-144077388 .  https://github.com/notifications/beacon/AAekDsQOlyKngZe62AYNNEVTcn72giVyks5o2peygaJpZM4GFru6.gif

Hey thanks for the log and the information, it is detailed. However, sorry for not being clear enough, I meant the XMPP log. The openfire java log is not quite useful usually.
The steps in order to have the Openfire XMPP logs are:

install Openfire Debugger plugin, http://www.igniterealtime.org/projects/openfire/plugins.jsp (From the plugin admin menu)
Enable it under Server/Server Manager/System Properties: plugin.debugger.interpretedAllowed -- true to print XML packets after they were parsed by the server. . The default value is false.

In my installation XMPP logs are accessible after:

sudo -i
cd /opt/openfire/logs. (actual location is configuration option of the server under the key xmpp.audit.logdir )
tail -f nohup.out

From my experience at some point, with any library these are going to be needed, so it is a step worth taking!
In either case I would not bet on being the certificate. S22.Xmpp/Sharp.Xmpp have a callback, when the certificate is not signed, but it must be explicitly be implemented.
By the way what is the value of your  xmpp.domain configuration property in open fire? It should be the XMPP domain. I can see this message on your logs:
"Unexpected answer in validation from: thor.manne.eu id: bc6e5a19 for domain: manne.eu"
and

Unexpected answer in validation from: thor.manne.eu id: bc6e5a19 for domain: manne.eu answer:<stream:error xmlns:stream="http://etherx.jabber.org/streams"></stream:error>

xmpp.domain should be manne.eu, if it is not please try just for testing's sake to change it  to  thor.manne.eu.

Hi !
The value for the domain is "manne.eu", which is correct.
I was a moment mooney and trusted you too much ... ;-)
I put "thor.manne.eu" into the domain and saw the dead.
I was no longer able to logon, because Openfire takes
all users relative to the registered domain, like<EMAIL_ADDRESS>Now, no user was valid :-(
I hacked around a lot until I remembered, that this also
at installtime happened to me ... ;-)
The xmldebugger plugin cannot be installed, because it needs
a newer server version.

The box is 15 years old and a plan to replace lies on my table since
two years and the new box too - a little old now ... I'll need debian
jessie with a new mono version and more power, but stuck in other
projects. Hope, before the end of the year, everything is on modern
software ...
I'll keep your new lib in memory, would even require a chatroom
feature, so to wait is the right moment.
Thanks for your investigation!
Best regards,
Manfred
From: PGS<EMAIL_ADDRESS>Sent: Tuesday, September 29, 2015 9:29 PM
To: pgstath/Sharp.Xmpp
Cc: mabra
Subject: Re: [Sharp.Xmpp] Session establishment failed (#9)
Hey thanks for the log and the information, it is detailed. However, sorry for not being clear enough, I meant the XMPP log. The openfire java log is not quite useful usually.
The steps in order to have the Openfire XMPP logs are:

install Openfire Debugger plugin, http://www.igniterealtime.org/projects/openfire/plugins.jsp (From the plugin admin menu)
Enable it under Server/Server Manager/System Properties: plugin.debugger.interpretedAllowed -- true to print XML packets after they were parsed by the server. . The default value is false.
In my installation XMPP logs are accessible after:
sudo -i
cd /opt/openfire/logs. (actual location is configuration option of the server under the key xmpp.audit.logdir )
tail -f nohup.out
From my experience at some point, with any library these are going to be needed, so it is a step worth taking!
In either case I would not bet on being the certificate. S22.Xmpp/Sharp.Xmpp have a callback, when the certificate is not signed, but it must be explicitly be implemented.
By the way what is the value of your xmpp.domain configuration property in open fire? It should be the XMPP domain. I can see this message on your logs:
"Unexpected answer in validation from: thor.manne.eu id: bc6e5a19 for domain: manne.eu"
and

Unexpected answer in validation from: thor.manne.eu id: bc6e5a19 for domain: manne.eu answer:/stream:error
xmpp.domain should be manne.eu, if it is not please try just for testing's sake to change it to thor.manne.eu.
—
Reply to this email directly or view it on GitHub https://github.com/pgstath/Sharp.Xmpp/issues/9#issuecomment-144164381 .  https://github.com/notifications/beacon/AAekDjEqyVLIKA6XhqUlAz5HEJ29XMknks5o2t4YgaJpZM4GFru6.gif

OK thanks Mabra. One thing that I must definitely put in the library is the ability to printout the received XML messages, this would be helpful for this situation as well.


Authentication not working on android 4.4

Hello,
I'm trying to figure out what is going on because I cannot login using Twitter only on android 4.4 (it works on android 6 and I can't test on 5 right now but it was working a few days ago)
After investigating a lot what's happening is that the TwitterConnect.onActivityResult method is never called after the user authenticate.
It is called once when all is initialized by cordova to "reset the activity callbacks" but never when I am done authenticating.
I put some breakpoints to figure out what was going on and it passes through OAuthActivity.onComplete in Twitter SDK so that should trigger the onActivityResult from what I understand.
The Android 6 phones uses SSO authent but I somehow forced them to use OAuth and it worked fine.
It also works fine on iOs.
Thank you for any help on that matter

Something I just realized is that the TwitterConnect.onActivityResult triggered by the reset of callbacks on android 4.4 is not called on android 6 which might be a clue
And I tested on versions .5.0 and 0.6.0 of the plugin

Hmm... That's really odd.
This plugin was developed specifically for an app targeting 4.4 (the latest at the time).
If I get some free time, I'll try to see if I can reproduce the problem.
Thanks for your efforts to report as much information as possible!

Ok I spent a bit more time and trying with a clean app I could pinpoint the problem to the AndroidLaunchMode
I was in singleInstance mode and switching to anything else fixes the problem so I switched to singleTask

It also fixed the same problem with Facebook plugin that I saw later.


PR: Pass version kwarg in every call to saveState/restoreState (Main Window)

Description of Changes
This is a followup to PR #14957. Unfortunately, in that PR I didn't set version in all calls of those methods, which broke custom layouts and restoring the main window state saved in previous sessions.
Issue(s) Resolved

Fixes #14962.
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: @ccordoba12

Thanks a lot @dalthviz for the suggestion! It's a very good idea! I implemented it in my last commits.

I inserted "version=1" into each call to saveState() and restoreState() and amazingly, it worked! Now Spyder 4.2.4 will remember the window size and position as in previous versions. Thanks for what turned out to be an easy fix, and for everyone's perceptiveness in recognizing where the problem was located.

Thanks for the confirmation @SFMornay! I'll release 4.2.5 with this fix very soon.


external login with google and twitter

How to use Google and Twitter to log in?
In the original IdentityServer4, we can easily add the following code to do it:
services.AddAuthentication()
.AddGoogle(googleOptions => { ... })
.AddTwitter(twitterOptions => { ... });
Thanks
Jack

Hi @jack1232
you can add another external providers here:
https://github.com/skoruba/IdentityServer4.Admin/blob/master/src/Skoruba.IdentityServer4.STS.Identity/Helpers/StartupHelpers.cs#L307
Thank you

I am closing this, feel free reopen. Thank you.


Bookstore - Connect redux-react completed

I have completed the connection of redux to react in the Bookstore project.

Implementation was on branch connect-redux-react
Gitflow observed
Static book items are received from the slice, and displayed in a reusable component
The App is wrapped with  component from react-redux
The store is passed into the
Dispatch actions was implemented using useDispatch
Button added with event handler for adding a book with id, title and author
Button also added to an event handler to remove a book by id.
Linters checked and passed.

Thanks


Release proposal: v5.0.2

Maybe next week and the changelog is likely to change prior to release (will update this OP if it does).

[2761afbf73] - build,test: add duplicate symbol test (Gabriel Schulhof) #1689
[f952b08f84] - gyp: move from __future__ import to the top of the file (cclauss) #1789
[4f4a677dfa] - gyp: use different default compiler for z/OS (Shuowang (Wayne) Zhang) #1768
[611bc3c89f] - lib: add .json suffix for explicit require (Rod Vagg) #1787
[d3478d7b0b] - meta: add to .gitignore (Refael Ackermann) #1573
[ec2eb44a30] - test: use Nan in duplicate_symbols (Gabriel Schulhof) #1689
[1597c84aad] - test: use Travis CI to run tests on every pull request (cclauss) #1752
[dd9bf929ac] - zos: update compiler options (Shuowang (Wayne) Zhang) #1768

CI: https://ci.nodejs.org/view/All/job/nodegyp-test-pull-request/132/

rebased with latest, updated OP with new commit list.
https://ci.nodejs.org/view/All/job/nodegyp-test-pull-request/134/
I might get this out in the next day or two. Then once it's out I'd like to merge #1794 which introduces more rigorous linting but shuffles a lot of code in the process, making merging and rebasing more difficult. Beyond that I'd also like to do some general refactoring, more code shuffling, to aid with maintainability. I just don't want to make existing PRs more difficult to land so it'd be nice to get some more things merged before I do that.

This is a nice milestone because with 189 days until Python 2 end of life, we are "syntax compatible" with Python 3.  That does not mean that the port to Python 3 is complete but it does allow for experimentation.  Thanks team!

released, also included 7a9a038 'test: add parallel test runs on macOS and Windows'

@rvagg shouldn't this be published in npm?

Yes, it should be, but it's not. I'm sure I published it! I have a fixed workflow for publishes. Maybe it errored during 2fa or something and I didn't notice cause I was in robot mode.
I've done a release now and it went through fine. Thanks for picking this up @joaocgreis.


Remove malware infected link

Solves #5

Many thanks for picking this up! Think I may just remove the whole link.


Change Proposal

Changed Volumes to ./containerdata/postgres and ./containerdata/redmine
Put everything into the docker-compose dir to have a complete set within one folder per project. :-)

The docker-compose.yml is a sample file. feel free to change your local copy as you see fit.


PaaS does not require serviceTypes

The fact that this doesn't break any tests is a worry. Currently we only explicitly test SCS services as that was the first that got implemented.

Did a casual diff of the schemas for G6 and G7 PaaS services, this seems to be the only material difference.
:+1:


500 server error for multiple requests with certificates

Hi guys,
I am writing a script that has 30 API's that is performing POST operations. So i am passing the headers and payload along with the certificates. each request has a keystore file and the corresponding passkey. When i try running my script, i get 200 OK for the first request but for the remaining requests I get a 500 server error. i dont think i am passing anything wrong because when i do them individually it gives me 200. i am new to ngrinder and dont have much idea about it. Can anyone provide a possible solution please?

can you share your script??