Call for High Quality Lightning Lectures - Community Sprint

🚀 Feature

Dear community,
As PyTorch Lightning mature, we believe it is important for the Lightning Team and its community to improve the Lightning onboarding process.
In that regards, the PyTorch Lightning Team (@Borda) built a https://github.com/PyTorchLightning/lightning-tutorials repo containing tuto-books (notebooks as python script). The tuto-books are converted into notebooks, fully tested on GPUs and injected directly within the Lightning documentation.
Learn more with this blogpost: https://devblog.pytorchlightning.ai/publishing-lightning-tutorials-cbea3eaa4b2c
The tutorials can be found there on the main page of the documentation: https://pytorch-lightning.readthedocs.io/en/latest/
@phlippe, PHD from the University of Amsterdam contributed their Deep Learning Course from 2020 and it is now rendered within Lightning documentation (released officially with v1.5). Great thanks from the entire community :)
However, the Lightning Team doesn't want to stop there and we would like to encourage the community to contribute more high quality course and collaborate closely with us to implement best practices while doing so.
We plan to organize a tutorial sprint to create high quality tutorial / course and would like the community to share ideas and collaborate with the Lighting Team in this project. We will soon share a Google Form for the community to share their ideas and possibly to contribute.
Here are the requirements for a tutorial to be accepted:

The Tutorial should present a field of AI in simple terms, not a single method. The UVA Course SSL or Meta-Learning notebooks are a great example of such works.

The Tutorial should contain high quality animated visuals, be engaging, well written and simple to follow. Assume new comers.

The tutorial should come with a short description, thumbnail.

The Tutorial should contain a YouTube video going through the notebook + attendees asking questions (Lightning Team and community). We might live stream the event and use the recording as its explanation ;)

Note: The Lightning Team Editors can help there with visuals, thumbnail
Here are some high quality notebooks we are planning to work on:
(This list isn't exhaustive and we would like the community to propose ideas based on their skills and interests).

[ ] Lightning introduction for engineers
[ ] Lightning introduction to distributed training and challenges. Ultimate guide.
[ ] Lightning best practices
[ ] Lightning introduction for researchers
[ ] Lightning and Video (PyTorch Video)
[ ] Lightning and PointCloud
[ ] Lightning and 3D Rendering (PyTorch3D)
[ ] Lightning and Quantum (Pennylane)
[ ] Lightning and Probabilistic Models (scvi-tools)
[ ]  Lightning and Forecasting (PyTorch Forecasting)
[ ] Lightning on Kaggle (Flash)
[ ] Lightning and 3D Images (Monai)
[ ] Lightning and Temporal Graphs (PyTorch Geometric / PyTorch Geometric Temporal)
[ ] Lightning and GANS
[ ] Lightning and Style Transfer
[ ] Lightning and Image Synthesis (https://github.com/CompVis/taming-transformers)
[ ] Lightning and Reinforcement Learning (Single / Multiple Agents)
[ ] Lightning and Online Learning
[ ] Lightning and Sequential Learning
[ ] Lightning and ODE
[ ] Lightning and Federated Learning
[ ] Lightning and Multi Tasks
[ ] Lightning and HPO (Optuna)
[ ] Lightning and Precision Scheduling
[ ] Lightning and Arts Generation !

# Applications related

[ ] PyTorch Profiler + ORTModule + DeepSpeed Sparse + OnnxRuntime.js + Triton (Exploration for optimized application building)

Bonus: Notebook translation would be a massive plus and would help to democratize AI in the entire world :)

Chinese
German
Czech
Hebrew
French
Russian
Spanish
Japanase
Hindi

If you are interested, please add a comment below and assign your name to the proposal(s), your level, share about your availabilities and your confidence into your contribution being impactful from 1 to 5 :)
Please, apply to this Google Form

If you enjoy Lightning, check out our other projects! ⚡

Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

I am a native Spanish speaker and I would be glad to help with the translation in that language.
Should I start somewhere in particular?

@tchaton
This list needs definitly also some lightning + huggingface transformers parts ( for example how to fine tune huggingface models with lightning)  🤗
wdyt?

I'm sorry, but I don't quite understand what would be the necessary steps to contribute to this repository with a translation:
from what I understand, it would be helpful to have the notebooks translated, but are there any of them already being worked on? If not, where should I upload the ones I translate?
I have looked around a bit (mainly in this repository and in the tutorials one) but I haven't found any specific place where the translatied notebooks should be located.
If there are already some translations being done or maybe a person or a group working on them (even if they are not related to the Spanish one) it would be very helpful for me if you let me know about them to be able to work together or to take them as an example to improve my efficiency.

Dear @Carlosbogo,
We are still polishing our ideas around translation. We were thinking to use automatic translation and use a translator to refine them. @Borda Anything else to add?
Best,
T.C

Hey @yifuwang. Let's keep this issue pinned, so new comers can find this issue faster and propose their contributions

I can imagine doing Lightning + Forecasting if it is up for grabs or collaboration in any sense.

Hello, @tchaton. I filled out the Google form and would like to contribute to the Lightning and Image Synthesis notebook. I've reviewed the attached repository, too, and maybe I can begin working on it or is it already assigned?

Hey @Ashish013,
That's great, yes please start working on it and share the PR in Lightning Slack for review.
Best,
T.C

Hello, I am the creator of the Proximal Policy Optimization (PPO) Implementation Deep Dive series. Pytorch-lightning is a fascinating project, and I hope to an RL tutorial using it sometime next year. Just posting here to keep track of it :)
cc @tchaton

Is there any interest in using Lightning for Music Generation?

Hey @mathemusician,
Yes, 100 %. If you are interested, you can start add an advanced example to Lightning Tutorials :)
Best,
T.C

Lightning is quickly becoming a standard part of my DL projects. I have several years of experience doing weakly supervised learning and would be interested in building out a tutorial along those lines for medical imaging. Open to collaboration too :)


[Feature request] I would like to trigger fivetran and airbyte sync from fal

New features requested:

Ability to specify fivetran and airbyte (EL) connections in profiles.yml
Magic functions for starting and checking sync operations

Specifying EL connections in profiles.yml
fal_test:
  target: dev
  fal:
    el:
      - type: fivetran
        api_key: my_fivetran_key
        api_secret: my_fivetran_secret
        connectors:
          - name: fivetran_connector_1
            id: id_fivetran_connector_1
          - name: fivetran_connector_2
            id: id_fivetran_connector_2
      - type: airbyte
        host: http://localhost:8001
        connections:
          - name: airbyte_connection_1
            id: id_airbyte_connection_1
          - name: airbyte_connection_2
            id: id_airbyte_connection_2
  outputs:
    dev:
      type: postgres
      host: localhost
      user: pguser
      password: pass
      port: 5432
      dbname: test
      schema: dbt_fal
      threads: 4

Magic functions for sync operations
# Starts airbyte sync and waits until it's completed
airbyte_sync(connection_id: str)
airbyte_sync_all()

# Starts fivtran sync and waits until it's completed
fivetran_sync(connector_id: str)
fivetran_sync_all()

These functions should also be available in FalDbt class:
from fal import FalDbt
faldbt = FalDbt(profiles_dir="~/.dbt", project_dir="../my_project")

faldbt.airbyte_sync_all()
# etc

Alternatively, we could have a combined sync function:
el_sync(el_type: str, connection_id: str, all: bool)
# el_type can be 'airbyte' or 'fivetran'
# connection_id and all are optional arguments, where either one or the other has to be provided.

faldbt.el_sync(el_type, connection_id)

🙏

Yes, connections are for airbyte and connectors are for fivetran. Unifying it is an interesting idea. We would have to teach the users what definition would be though.
I agree that being able to refer to connections by name would make it easier to work with them.

You could want several Airbyte connections. I think we have to name a profile, like dbt names profiles (e.g. dev in your example profiles.yml)
fal_test:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      user: pguser
      password: pass
      port: 5432
      dbname: test
      schema: dbt_fal
      threads: 4
  fal:
    el:
      my_fivetran:
        type: fivetran
        api_key: my_fivetran_key
        api_secret: my_fivetran_secret
        connectors:
          - name: fivetran_connector_1
            id: id_fivetran_connector_1
          - name: fivetran_connector_2
            id: id_fivetran_connector_2
      my_airbyte:
        type: airbyte
        host: http://localhost:8001
        connections:
          - name: airbyte_connection_1
            id: id_airbyte_connection_1
      my_other_airbyte:
        type: airbyte
        host: http://localhost:8002
        connections:
          - name: airbyte_connection_2
            id: id_airbyte_connection_2

This also makes me wonder about the need of the el property between fal and my_fivetran. Why would we need this and couldn't just list my_fivetran directly below fal?

Great point about multiple Airbyte connections! But before implementing it, I would wait until a related issue is raised by a user actually needing that.

But that means when asked about it we would release a breaking change. I think that matches the current dbt outputs approach and we want to probably be as close as possible to that.
This would change functions a bit:
el_sync(target='my_airbyte', opts={'connection': 'airbyte_connection_2'})

Or:
airbyte_sync(target='my_airbyte', connection='airbyte_connection_2')

I am not inclined to any of the el_sync vs airbyte_sync yet. I may like airbyte_sync more to be able to specialize it as needed per possible EL tool; these 2 match the patterns a lot, but that not may hold for future integrations.

Let's call "fal" -> "fal_extract_load" and get rid of the "el" key.

Let's update el_sync function to remove target as an arg, and add name that corresponds to the key name of the connection.


It is now possible to proxy a SOAP endpoint

Possible solution to authenticating when using a SOAP endpoint.  Example proxy.config:
<?xml version="1.0" encoding="utf-8" ?>
<ProxyConfig>
    <serverUrls>
        <serverUrl url="https://server.com:6443/arcgis" matchAll="true" username="siteadmin" password="********" />
    </serverUrls>
</ProxyConfig>

Example usage:
https://server.com/proxy/proxy.ashx?https://server.com:6443/arcgis/services/QA/QA/MapServer

Hi @ca0v - thank you for the pull request. Did you test this with a server after changing it to enable tokens thru GET?
Since you mentioned http://enterprise.arcgis.com/en/server/latest/administer/linux/enable-token-acquisition-through-an-http-get-request.htm in your comment on #498 ...

I just set these settings to "false" and I am still able to get a token via querystring credentials  I'm on 10.6.  I deleted and tried again:
{
  "type": "BUILTIN",
  "properties": {
    "sharedKey": "*",
    "longTimeout": "1440",
    "shortTimeout": "60",
    "allowHttpPostQueryParams": "false",
    "allowHttpGet": "false"
  }
}

2018-08-17 14:37:51  Adds support for WSDL-based endpoint
2018-08-17 14:37:51  Querying security endpoint...
2018-08-17 14:37:51 Sending GET request: https://.../arcgis/rest/info?f=json
2018-08-17 14:37:51  Service is secured by https://.../arcgis/tokens/: getting new token...
2018-08-17 14:37:51 Sending POST request: https://.../arcgis/tokens/?f=json&request=getToken&referer=http://localhost/proxy/proxy.ashx&expiration=60&username=***&password=***
2018-08-17 14:37:51  Token obtained: DMV***.

So I'm not sure what's up with that.  Seems like once you turn it on it stays on.

I don't think it's currently possible use this proxy to access secure SOAP services.  Is there an ETA on getting this request pulled or the functionality otherwise made available?


add Timeout config for ClusterAdmin related Requests

This is the fix for the issue reported here #1120

Thanks!


Add Minecraft Purpur Support

We already have a purpur template here
Is there any reason for having a second one instead of updating the existing one?


Feature: Allow BTDF to be used on a build server without any installation

In many cases a build server is off limits to developers, so when a new version of the BTDF comes along it requires having an admin uninstall the old version and install the new version on the build server.  Or, a new version comes along, developers begin to use it but forget to update the build server, thus the build server-generated MSI's don't match the developers' intentions.
Add the ability to override the DeploymentFrameworkRootDir property in a .btdfproj and direct the project to use BTDF files from any chosen directory, such as a location checked into source control.  This can avoid ever installing the Framework on the build server in the first place.
This work item was migrated from CodePlex
CodePlex work item ID: '9961'
Assigned to: 'tfabraham'
Vote count: '1'

[UnknownUser@5/16/2013]

[UnknownUser@3/14/2013]
Resolved with changeset 72198.


update deps and cross compile

I just added scala 2.13  support, but I removed the Seq.view methods as they raised type error. I'm not sure about the impact of the views, so might worth correct the ViewSeq error instead

Thanks for the contribution!
The idea of using view was to prevent extra collection allocations when converting into a Python value. This could probably be fixed by having createList take an Iterable, so then using .view will compile under 2.13.

Closing, support for Scala 2.13 was merged with #46


[FLINK-6760][table]Fix OverWindowTest alias test error

[x] General

The pull request references the related JIRA issue ("[FLINK-6760][table]Fix OverWindowTest alias test error")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[ ] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[ ] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

Thanks @sunjincheng121, I will merge this.


[BUG] Scaled down icons have darker borders

:information_source: Info
3.16.4
:speech_balloon: Description
The generated icons have a dark border instead of fading to transparent.
On small images (like 44x44) this is very noticable.
Original:

Start Menu:

I opened the generated icon in Gimp and put a white background to see if it is dark or transparent, apparently, this is dark.

:scroll: Pubspec.yaml
msix_config:
  display_name: LocalSend
  publisher_display_name: Tien Do Nam
  identity_name: 11157TienDoNam.LocalSend
  logo_path: assets\packaging\windows-icon-512.png
  languages: en, ar, bn, cs, de, es-ES, fa, fr-FR, hu, in, it, iw, ja, ko, ne, nl, pl, pt-BR, ru, sv, tr, uk, zh-CN, zh-HK, zh-TW
  os_min_version: 10.0.19041.0

  startup_task:
    task_id: localsend
    enabled: false
    parameters: autostart

Better?

ℹ Info
3.16.4
💬 Description
The generated icons have a dark border instead of fading to transparent.
On small images (like 44x44) this is very noticable.
Original:

Start Menu:

I opened the generated icon in Gimp and put a white background to see if it is dark or transparent, apparently, this is dark.

📜 Pubspec.yaml
msix_config:
  display_name: LocalSend
  publisher_display_name: Tien Do Nam
  identity_name: 11157TienDoNam.LocalSend
  logo_path: assets\packaging\windows-icon-512.png
  languages: en, ar, bn, cs, de, es-ES, fa, fr-FR, hu, in, it, iw, ja, ko, ne, nl, pl, pt-BR, ru, sv, tr, uk, zh-CN, zh-HK, zh-TW
  os_min_version: 10.0.19041.0

  startup_task:
    task_id: localsend
    enabled: false
    parameters: autostart

Did you use a white background for the icon just for a more vivid demonstration of the issue, or are you actually planning to add a circular white background to the icon? Because if that's the case, it's not a very good idea.


Platform log data retained for 12 months in active state and 18 months in inactive state.

In order to comply with EO M-21-31, we want platform log data retained for 12 months in active state and 18 months in in-active state.
Background

Presidential Memorandum M-21-31.
Notes from discussion with assurance team about M-21-31

Acceptance Criteria

[ ] Inspecting config confirms that platform logs are retained for expected retention period

Security considerations
Changes to retention period will allow us to comply with EO M-21-31
Background
We are treating storage in ELK as "active" storage and storage in S3 as "inactive" storage.
Platform logs are currently stored in ELK for 30 days.
Implementation sketch

[x] Update configuration for ELK to store platform logs for 365 days
[x] Update S3 logs to be retained for 30 months: https://github.com/cloud-gov/cg-provision/blob/e2f1461fefa3ab01a608da8dde4ce68b76000486/terraform/modules/logsearch/platform_log_bucket.tf#L2-L25

All PRs have been merged, closing


How to calculate accuracy of (un)corrected reads?

Hi @thackl,

Depending on your proovread paper, how to calculate the accuracy of (un)corrected reads?

Following your suggestion, I use mapPacBio.sh of BBMap to evaluate the correction result. When using mapPacBio.sh to map corrected reads to reference, the length of some reads exceeds the maxlength threshold(6000) of BBMap. What is your solution?

Best
Weilong

Accuracy simply is "number of matching bases" / "number of total read bases". We use read length rather than alignment length/aligned read length, as we do also want to penalize clipped ends, because clipping indicates erroneous read ends and ignoring clipped bases would overestimate accuracy. Regarding uncorrected reads, we used to placement information and the chimera status of the corrected read to align the uncorrected read with high sensitivity to the same reference location. For the paper, we ignored chimeric reads, because aligning those globally will only give proper results, if the reads is first cut at break points. This, however, is non trivial and we did not have particular interest in the specific error rate of the few chimeric reads - assuming, the accuracy won't significantly differ from other uncorrected reads

Use reads in FASTA format, bbmap will internally split them in 6kb chunks and will therefore be able to map reads of any length. There is a slight chance, that the split read mapping might differ from aligning the read as a whole, but I have not yet encountered such a scenario especially for corrected reads.

Thank you for your response.


TSEMMG test cause the MUX to crash

Im simulating an EMM towards a commercial MUX/Head-end using tsemmg for a STB test bed. In exactly 2 hrs the EMM stream cause the mux to crash with no further info.
./tsemmg --mux <IP_ADDRESS>:5000 -v --log-protocol --log-data --client-id 0x00100001

Scenario 1 - tseemg without xml file sending dummy emm data
Scenario 2 - tsemmg with a simple xml file as below containing two lines
Scenario 3 - same file with --section-mode

<tsduck>
<generic_short_table table_id="0x82">
  AA BB CC DD EE FF AA BB AA 01 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11
</generic_short_table>
<generic_short_table table_id="0x82">
  BB CC DD EE FF AA BB CC AA 02 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22 22
</generic_short_table>
</tsduck>

All 3 scenarios result the same identical mux crash in exactly 2 hours. Can this be a fault in MUX or do we need any extra parameters to send with emm.
Would highly appreciate any hint.

Hi @nikpakar
Does the MUX equipment actually crash or does it simply close the connection with the EMMG? A commercial production equipment, MUX or whatever, shall never crash. Even in front of a faulty peer equipment, it should survive. A crash is a bug, there is no room for arguentation. Now, how to characterize it?
Since you use --log-protocol --log-data, did you see any abnormal  message from the MUX? Any stream_error message? Maybe you can try  --log-protocol=info --log-data=debug to avoid the flood of data_provision messages, to concentrate on the protocol itself. Be sure to redirect standard output and standard error to a text file to be able to analyze it afterwards.
tsemmg is just a test tool. It tries to respect the bandwidth which was actually allocated by the MUX. But, after a couple of hours, maybe the amount of data is slightly larger than the allocated bandwidth. Normally, the MUX shall reply with a stream_error message with an exceeded bandwidth error (0x000F). But there is maybe a bug in the MUX in case of input overflow (which would be a critical bug for a production equipment).
To stress the capability of the MUX to resist to overflow, you may try --bandwidth XXX --ignore-allocated with various XXX bandwidth. You can see the stream_BW_request and stream_BW_allocation response. At some point, you will ask for too much and the returned allocation will be less than the requested value. The option --ignore-allocated will then force tsemmg to ignore the allocation and overflow the MUX with the initially requested EMM bandwidth. This will show you how the MUX behaves in case of overflow. If it crashes immediately, there is a bug in the MUX and your initial crash is maybe due to a slow drift in tsemmg output. But if the MUX behaves properly, sends a stream_error message with an exceeded bandwidth error and drop EMM's in excess, then your initial problem is probably elsewhere.

Hi Thierry,
Many thanks for your detailed reply. Below are the observations with your suggestions.

Initial crash identification - tsemmg just lost the connection, and the MUX gone to non responsive mode stopping multiplexing and output as well and had to restart to bring it back. Hence identified as a sort of crash
At the point of crash MUX's terminal frozen and shows Data Provision B/W as 2900 Kb/s where it was normally around 100 Kb/s. (MUX maximum allocated is 300 Kb/s while tseemg runs with 100 Kb/s
Ran with protocol=info and data=debug for more clarity with different bandwidth with --ignore-allocated. After 300 Kb/s, I can see MUX properly sending error message "error_status = 0x000F (exceeded bandwidth)"
When reach 2000 Kb/s MUX crashes

So the conclusion is with the running time, tsemmg build some extra BW and the mux continue to give error 0x000F till it reach around 2000 Kb/s but crashes afterward.
Question: Is there anyway we can handle this bandwidth exceed from tsemmg ? i.e. forcing it not to exceed certain.
I will also try set MUX's max limit to 3000 Kb/s to see how well it can tolerate and whether tsemmg goes further beyond that as well. Its only 2 lines of emm data its sending for the test bed.
Once again many thanks for your detailed response and would highly appreciate any further advise.

"Normally" (a mythic adverb in computing), tsemmg sends fake EMM's at a fixed bitrate. This bitrate is initially given by the --bandwidth option (100 kb/s by default) and is reduced when the stream_BW_allocation response message from the MUX allocates less. The option --ignore-allocated is here to test the capabilities of the MUX and stays with the --bandwidth value, even if the MUX allocates less.
But, in all cases, the data should be sent at a fixed bitrate, all the time. What you see on the MUX is that the received EMM bandwidth was 100 kb/s for a while and then increases to 2900 kb/s. The question is now: is the MUX right or wrong? Does tsemmg suddenly go crazy and multiply its output speed by 29? Or does the MUX incorrectly evaluate the input bandwidth?
As usual, the response is on the wire. I suggest that you retry your original test and capture packets in parallel using wireshark (use a capture filter based on the MUX address and port). After the MUX crashes, it should be relatively easy to check the time between two TCP packets using wireshark. If the EMM bandwidth jumps from 100 to 2900 kb/s, the duration between packet time stamps should be divided by 29. You should be able to see that quite easily on wireshark GUI. Take care that TCP packets may be sent in bursts. So, you have to identify the groups of TCP packets (very short interval between them) and the interval between them.
If you can confirm that the EMM bandwidth actually jumps to 2900 kb/s, this is a bug in tsemmg and I will investigate (there is also a bug in the MUX which does not behave properly).

Thank you so much again for your clarification and got your point.
Im running the first case now with tcpdump on the interface. Will analyse the capture from wireshark and post the result.

I have captured the crash point and struggling a bit to identify the point as you have described. Timestamps seems quite closer in every packet. Below is the crash point on Wireshark. For some reason Wireshark doesn't decode them as simulcrypt even the protocol is enabled.
tsemmg - <IP_ADDRESS>
mux - <IP_ADDRESS>
60041 2920.845249    <IP_ADDRESS>       <IP_ADDRESS>        RSL Cell Broadcast 1570   SMS BroadCast CoMmanD  (CBS Page)
60042 2920.845252    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2974   42756 → 5000 [PSH, ACK] Seq=43343132 Ack=84 Win=64066 Len=2920
60043 2920.845276    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   BCCH INFOrmation
60045 2920.845473    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   BCCH INFOrmation
60046 2920.845480    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   DATA INDication
60048 2920.845697    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   42756 → 5000 [ACK] Seq=43351892 Ack=84 Win=64066 Len=1460
60050 2920.845912    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   BCCH INFOrmation
60052 2920.846197    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   DATA INDication (DTAP) (BCC)
60054 2920.846400    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   ESTablish CONFirm
60056 2920.846685    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2350   [Malformed Packet]
60058 2920.846909    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   42756 → 5000 [ACK] Seq=43361488 Ack=84 Win=64066 Len=1460
60060 2920.847149    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   [Malformed Packet]
60062 2920.847467    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   unknown 0
60064 2920.847666    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2974   42756 → 5000 [PSH, ACK] Seq=43365868 Ack=84 Win=64066 Len=2920
60066 2920.847935    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   ip.access CRCX
60068 2920.848234    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2994   42756 → 5000 [PSH, ACK] Seq=43370248 Ack=84 Win=64066 Len=2940
60070 2920.848472    <IP_ADDRESS>       <IP_ADDRESS>        TCP      658    42756 → 5000 [ACK] Seq=43373188 Ack=84 Win=64066 Len=604
60072 2920.848718    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   MultiRate CODEC MOD PERformed
60074 2920.848973    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   [Malformed Packet]
60076 2920.849265    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2254   CCCH LOAD INDication
60078 2920.849500    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   MultiRate CODEC MOD PERformed
60080 2920.849888    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2614   BCCH INFOrmation
60081 2921.065900    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60082 2921.505923    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60083 2922.369970    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60085 2924.097922    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60088 2927.681933    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60107 2934.593917    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460

I have captured the crash point and struggling a bit to identify the point as you have described. Timestamps seems quite closer in every packet. Below is the crash point on Wireshark.  The protocol is however correct, otherwise you wouldn't be able to communicate with the MUX. Did you explicitly enable DVB SimulCrypt in Wireshark? I don't think it will detect it automatically.
There are some interesting information in you dump. As I predicted in a previous post, you get bursts of 20 to 30 IP packets, with a 200 ms gap between bursts.
59939 2920.242661    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   42756 → 5000 [ACK] Seq=43209496 Ack=1 Win=64149 Len=1460
59941 2920.243052    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2102   42756 → 5000 [PSH, ACK] Seq=43210956 Ack=1 Win=64149 Len=2048
-- 200 ms gap
59943 2920.434770    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2974   42756 → 5000 [PSH, ACK] Seq=43213004 Ack=84 Win=64066 Len=2920
59944 2920.434778    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   unknown 0
.... more messages
59995 2920.440540    <IP_ADDRESS>       <IP_ADDRESS>        RSL      3126   BCCH INFOrmation
59997 2920.441042    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1398   unknown 125
-- 200 ms gap
59999 2920.640898    <IP_ADDRESS>       <IP_ADDRESS>        RSL Cell Broadcast 2974   SMS BroadCast CoMmanD  (CBS Page)
60000 2920.640907    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   BCCH INFOrmation
.... more messages
60031 2920.644478    <IP_ADDRESS>       <IP_ADDRESS>        RSL Cell Broadcast 1514   SMS BroadCast CoMmanD  (CBS Page)
60033 2920.644794    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2614   [Malformed Packet]
-- 200 ms gap
60036 2920.844904    <IP_ADDRESS>       <IP_ADDRESS>        TCP      2974   42756 → 5000 [PSH, ACK] Seq=43327016 Ack=84 Win=64066 Len=2920
60037 2920.844912    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2974   unknown 0
.... more messages
60078 2920.849500    <IP_ADDRESS>       <IP_ADDRESS>        RSL      1514   MultiRate CODEC MOD PERformed
60080 2920.849888    <IP_ADDRESS>       <IP_ADDRESS>        RSL      2614   BCCH INFOrmation
-- 200 ms gap
60081 2921.065900    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460
60082 2921.505923    <IP_ADDRESS>       <IP_ADDRESS>        TCP      1514   [TCP Retransmission] 42756 → 5000 [ACK] Seq=43381832 Ack=84 Win=64066 Len=1460

First observation, the session ends with endless TCP retransmissions at the same TCP sequence number. This means that the MUX is hung either at process or operating system level and no longer reads packets. The MUX "application process" did not crash, otherwise the OS of the MUX would have probably closed the socket, which is not the case (no TCP RST). There is clearly a bug in the MUX.
Second, this dump is the end of the transmission. Have a look at the beginning of the session, after the setup, or in the middle of the session, when EMM's are sent on a nominal bitrate. Do you observe the same patterns of 20-30 messages with 200 ms gap?

If yes, then the transmission speed has not significantly changed, and certainly not by a factor of 29. So, the jump to 2900 kb/s is an interpretation error from the MUX which seems to be otherwise confused anyway. You have to report the issue to the MUX vendor, it is on their side.
If no (just one or two message per burst or much longer gaps), then tsemmg went crazy. I will have to investigate. You may also report to the MUX vendor that their device hangs in the middle of operation but they will probably respond that the operational conditions are not nominal, (which is not an excuse to crash a production system).

Please let us know the result.

Thank you so much again for the investigation.
Simulcrypt is enabled in wareshark, but didn't decode. I have specifically enabled it while disabling all others. Still didnt worked. Wireshark is the latest downloaded from the site yesterday.
The trace was on for some time even after the crash and the 6 retransmission we see are the only ones at the end. This is probably the MUX process initially hung and immediately died after few seconds. There are no comms afterward. and mux was not responsive at all.
There are no such 200ms gap at the very beginning, but can see towards the end after around 2 - 2 1/2 hrs.
MUX vendor has investigated and identified that we are sending larger bursts often at the later stage and its not that mux crashed from the first burst, but after many. and the bursts are not exactly 2900, they are varied from 1800 - 2900 kb/s most.
They have also confirmed the mux has crashed due to quite a few bursts it reached which are much larger than it set for. Mux is set to accept 300 Kb/s.

They have also confirmed the mux has crashed due to quite a few bursts it reached which are much larger than it set for. Mux is set to accept 300 Kb/s.

There is a confusion between "burst" and "bitrate". A bitrate is never smooth, we never send bits or even packets one by one. There is always some kind of burst. The actual question is how many bits over which period? In other words, what is the duration of a so-called burst?
The EMMG/PDS <=> MUX protocol does not specify the granularity of the burst. This is implementation dependent. The protocol only specify bitrates and a bitrate is an average, a number of kb over one second. So, having bursts is normal. They just need to be "reasonable".

the bursts are not exactly 2900, they are varied from 1800 - 2900 kb/s most.

What does this mean exactly? Does this mean that 2900 kb are received every second? In that case, the average bandwidth is 2900 kb/s, which is incorrect. Or does this mean that 2900 kb are received in one second and then nothing during 28 seconds?

I think I have misinterpreted the word by saying a burst. My mistake, extremely sorry about it.
What they say is they see higher bitrates ranging from 1800 to 2900 Kb/s from the mux.
If its ok I can post two screenshots from the mux terminal which shows when its normal and when its receiving higher.

What they say is they see higher bitrates ranging from 1800 to 2900 Kb/s from the mux.

In that case, we are back to the initial question: is the MUX right or wrong when it evaluates the bitrate?
Could you provide two pcap files from your Wireshark capture? The idea is to get a 5-second capture of the middle of the session (when the bitrate is considered normal) and the last 5 seconds before the MUX hangs.
If tsemmg goes crazy with bitrates at some point, could you give the exact TSDuck version and operating system type and version, as well as if this is a virtual or physical machine?

Thank you so much again. Im preparing to isolate the network and run the test again and will capture those 5 seconds and send it.
Meanwhile, TSDuck is the latest version from git runs on Ubuntu 20.04 in a LXC container. LXC container runs on a Proxmox PVE. Would that be the reason for such behaviour ?

LXC container runs on a Proxmox PVE. Would that be the reason for such behaviour ?

It could be. In fact, I had this in mind when I asked the question. I have no experience with Promox PVE but I have quite a lot with VMware on a macOS host and some with VirtualBox on a Windows host. In short, real-time response and time precision are not so good on virtual machines. This is my experience but specific tuning methods may exist.
As an example, there is a unitary test program with TSDuck in subdirectory src/utest. It tests all the low-level features of most base C++ classes in TSDuck. All the classes which encapsulate system time features are tested (timers, monotonic clock, synchronization timeouts, etc). It is difficult to test time in an automatic and reproducible manner. So, there are some precautions. For instance, a 500 ms timeout tests succeeds if the difference of measured times is greater than 500 ms and less than 500 ms + a reasonable margin (say +400 ms to be generous). The same cautious approach is used everywhere for tests involving time. On physical machines, macOS, Linux, Windows, all tests always pass. On virtual machines, however, some of these tests fail from time to time. For instance, a 500 ms timeout is woken up after 450 ms (or 450 measured ms). Some other timers are much longer than expected. I do not know if the problem comes from the action (timer, timeout) or its measurement but in all cases there is some inconsistency in the system time management of the virtual machine. The failure rate is not very high, say less than 5 or 10%, but exclusively on virtual machines, never on physical machines. So, I had to create a "weak failure" status and a different class of unitary test assertions for tests involving timers. If the unitary test suite completes with weak failures only, all those failures are reported but the overall test suite is considered successful. Without that class of unitary test weak failures, automating the build on many virtual machines was close to impossible, too many failures were reported.
I think that have seen that on Linux VM's only but I have only one Windows VM, no macOS VM (just a real Mac) but more than 10 Linux VM's because of the large number of different distros. One interesting point: On GitHub, all push actions trigger an automatic compilation, unitary test suite and system test suite on Linux, Windows and macOS. All of these run in GitHub-hosted virtual environment. I am notified only in case of failure, so weak failures are unnoticed unless I explicitly examine the log. I rarely examine the logs of successful CI/CD operations but I do from time to time and I noticed that kind of time-related weak-failures sometimes. So, yes, virtual machines matter.
Containers use the host kernel, so they should be less impacted, at least if the kernel runs on a physical machine. Virtual machine + containers seem a bad idea to run time-critical systems. But I may be wrong and others may have some better tuning experience of that environment.
As a conclusion, I would say that virtualization is a great IT tool for data processing servers, application servers, web servers, etc. but very poor for applications which require some time accuracy in the millisecond magnitude.

Thanks a lot again for the very comprehensive inputs. Indeed lots of valuable information. Thanks again.
My LXCs runs on the Proxmox and the proxmox runs on a bare-metal Sun Fire X4175. But i think it ll be really worth while to test it on a bare-metal server based on your inputs. So im setting up a new environment with a new hardware server.
Will run the same test from that and update the result first.

Result with a physical server was also same. Below are the captures from the start, at 1st hr, at 2nd hr and at the end and also tsemmg output.
tsemmg-pcap.zip

Thanks, I will have a look at your capture files.
For your information, Wireshark can format the DVB SimulCrypt protocol on this session. As mentioned in a previous post, you have to enable it on this specific TCP port because there is no standard port for this protocol (unlike SSH, FTP, etc) and it cannot be recognized automatically. Use Analyze -> Decode As... and then, considering that your MUX uses the server port 5000:

And the packets are displayed as follow:

Hi @nikpakar
I did a quick analysis tool for pcap files (based on the TSDuck pcap plugin code). It gives the data size (and bitrates based on timestamps) of the TCP data size. Filtering on tsemmg -> MUX, the data size includes the DVB SimulCrypt overhead (a few percents).
From your capture files:

In the "after 1 hour" file, the bitrate is 76.967 kb/s
In the "end" file, the bitrate is 94.356 kb/s

So, my conclusion is that the EMMG bandwidth consistently remains under 100 kb/s. Your MUX must have a severe bug somewhere with multiple consequences: incorrect input bitrate evaluation, hang, crash, etc.
The capture files prove that the incoming data are correct and the MUX is not. You may use them to report the bug to the MUX vendor. A link to this page would be probably useful as well.

Hi @lelegard
Thank you so much for the comprehensive analysis. I will take it up with the MUX vendor to further troubleshoot the issue.
Thanks about the tcp port mapping on Wireshark. It works now and decoding as simulcrypt.
Thank you so much again.

Hi @lelegard
After investigating, MUX vendor suggested to restrict number of datagrams in a tcp packet to 7. So I have used tsemmg's --bytes-per-send parameter to control the datagrams to 7.
Interestingly this changed the behaviour.
Now in exact time duration, tsemmg reports "tsemmg: too many invalid messages from mux:5000, disconnecting" in the log and exit.
MUX doesn't crash now and stays in good health.
When restart tsemmg, everything works fine again
This repeats every 2 1/2 hrs.
Wireshark now shows tsemmg sends few Malformed packets towards the end and the MUX replies with Stream error messages to the tsemmg just before the tsemmg exit with "too many invalid messages:"
Attached is the new capture at the exit point. Im troubleshooting further. If you see any thing out of this development so far, would highly appreciate your comment.
new-end.pcapng.zip

Hi @lelegard
After investigating, MUX vendor suggested to restrict number of datagrams in a tcp packet to 7. So I have used tsemmg's --bytes-per-send parameter to control the datagrams to 7.
Interestingly this changed the behaviour.
Now in exact time duration, tsemmg reports "tsemmg: too many invalid messages from mux:5000, disconnecting" in the log and exit.
MUX doesn't crash now and stays in good health.
When restart tsemmg, everything works fine again
This repeats every 2 1/2 hrs.
Wireshark now shows tsemmg sends few Malformed packets towards the end and the MUX replies with Stream error messages to the tsemmg just before the tsemmg exit with "too many invalid messages:"
Attached is the new capture at the start and exit point. It shows no malformed packets at the beginning but only appear at the end.
Im troubleshooting further. If you see any thing out of this development so far, would highly appreciate your comment.
New start & end.zip

Hi @lelegard
After investigating, MUX vendor suggested to restrict number of datagrams in a tcp packet to 7. So I have used tsemmg's --bytes-per-send parameter to control the datagrams to 7.
Interestingly this changed the behaviour.
Now in exact time duration, tsemmg reports "tsemmg: too many invalid messages from mux.., disconnecting" in the log and exit.
MUX doesn't crash now and stays in good health.
When restart tsemmg, everything works fine again
This repeats every 2 1/2 hrs.
Wireshark now shows the MUX replies with Stream error messages to the tsemmg just before the tsemmg exit with "too many invalid messages:" and error message says "Exceeded BW (peak 1949184bps) : forcing stream close"
MUX vendor confirms that they read this correct and this is the actual cause.
Attached is the new capture of the last 1+ hour till it exit with this error.
Im troubleshooting further. If you see any thing out of this development so far, would highly appreciate your comment.
5. New end.pcapng.zip

Hi @lelegard
I have tested the same scenario with a completely different MUX (Different model, software but same vendor)
And the result appeared to be identical. Were you able to identify any thing from my last findings.
Thanks again for kind support.

Thanks for the latest feedback. I am away for a couple of days without any computer. I will check by the end of the week.

Hi @nikpakar,
I analyzed the file "5. New end.pcapng" by ranges of captured packets. The TCP data bitrate from <IP_ADDRESS> to <IP_ADDRESS>:5000 is consistently 95-96 kb/s except in the end phase. After packet number 51600 (approximately), the data bitrate suddently jumps higher.
By ranges of 100 captured packets:
Packet 51000:     96,734 b/s
Packet 51100:     96,540 b/s
Packet 51200:     96,542 b/s
Packet 51300:     96,730 b/s
Packet 51400:     97,516 b/s
Packet 51500:     96,540 b/s
Packet 51600:    123,170 b/s
Packet 51700: 13,684,932 b/s
Packet 51800:  3,886,039 b/s
Packet 51900:  3,595,472 b/s
Packet 52000:  2,839,176 b/s
Packet 52100:  3,212,059 b/s
Packet 52200:  2,909,507 b/s
Packet 52300:  2,572,278 b/s

So, yes, there is a weird issue in tsemmg. I will investigate but it may be hard to reproduce.

Which version of TSDuck do you use?
What is the exact tsemmg command which was used for capture file "5. New end.pcapng"?
Could you please provide the end of the tsemmg log, including the "too many invalid messages from mux" (it shouldn't happen).

Hi @lelegard
Thanks again for investigating and looks like we have finally found the reason.
This jump is always consistence and it happens around that very same time after tsemmg starts. you can easily reproduce it by letting the tsemmg run for about 2 1/2 hrs. This always happens around that time.
Version
tsemmg: TSDuck - The MPEG Transport Stream Toolkit - version 3.28-2470
and this is the exact command i run now
./tsemmg -v --mux <IP_ADDRESS>:5000 --log-protocol --log-data --client-id 0x00FF0001 --section-mode --bytes-per-send 203 emm.xml
(bytes per send was introduced later, and  without that and --section-mode also the result was the same.)
End of tsemmg does not show any other logs or errors other than
tsemmg: too many invalid messages from mux.., disconnecting
just before it exit.

Hi @nikpakar,
I fixed the issue (or at least I think it is fixed). Could you please rebuild from the latest repo and check if the fix is correct in your case?

Hi @lelegard
Thank you so much for the fix. Im now running tseemg for 4hrs without that issue. It used to come in at 2 1/2hrs.
So looks like its fixed. Thanks a lot again and will keep it running for longer period and update the result again.

Hi @lelegard
Its been running for more than 24hrs now without the issue. We can confirm its been fixed now.


Array of complex data into FixedLengthRecord - how ?

`
[FixedLengthRecord(FixedMode.AllowLessChars)]
public class Customer
{
[FieldFixedLength(5)]
public int CustId;
[FieldFixedLength(30)]
[FieldTrim(TrimMode.Both)]
public string Name;

[FieldFixedLength(8)]
[FieldConverter(ConverterKind.DateMultiFormat, "ddMMyyyy", "MMyyyy")]
public DateTime AddedDate;

[???]
public Address[] Adresses
}`
`[FixedLengthRecord(FixedMode.AllowLessChars)]
public class Address
{
[FieldFixedLength(5)]
public int AddId;
[FieldFixedLength(30)]
public string City;

[FieldFixedLength(30)]
public string Street;

[FieldFixedLength(8)]
public string Number;

}`
What I should make into Attributes for this part of class (array of Address[] class)
[???]
public Address[] Adresses

I found this useful when I had the problem
https://github.com/MarcosMeli/FileHelpers/blob/ecc07e9e7942394f987af0cafd1ef6ca10a5ffe2/FileHelpers.Tests/Tests/Common/ArrayFields.cs#L536


add zenodo badge

not sure what to do about what's new right now.
closes #151

Does it automatically update? or should we use https://zenodo.org/record/3473593#.XZfNwvdlCis ?

not sure what yours actually links to. the "cite all versions" box says to use https://doi.org/10.5281/zenodo.3473592

mine links to the beta tagged version


[5.0.22-beta] Launch: Main class not found

I am trying to use this libarary to launch minecraft, but it throws this error:
Error: Could not find or load main class net.minecraft.client.main.Main
Caused by: java.lang.ClassNotFoundException: net.minecraft.client.main.Main

This is the full source code
import { Launcher, Version } from "ts-minecraft";

const javaPath = '/usr/bin/java';
const gamePath = 'temp';
const versionId = '1.14.2';

async function main() {
  const metas = await Version.updateVersionMeta();
  const version = metas.versions.find(version => version.id === versionId);
  await Version.install('client', version, gamePath);
  const process = await Launcher.launch({ gamePath, javaPath, version: version.id });
  process.stderr.on('data', data => console.error(data.toString()));
}

main().catch(console.error);

And this is the contents of my gamePath:
├── assets
│   ├── objects // IGNORED
│   └── indexes
│       └── 1.14.json
├── libraries // IGNORED
└── versions
    └── 1.14.2
        ├── 1.14.2.jar
        ├── 1.14.2.json
        └── 1.14.2-natives
            ├── libfliteWrapper.so
            ├── libglfw.so
            ├── libglfw_wayland.so
            ├── libjemalloc.so
            ├── liblwjgl_opengl.so
            ├── liblwjgl.so
            ├── liblwjgl_stb.so
            └── libopenal.so

Here are the spawnargs of the proccess:
[
  '/usr/bin/java',
  '-Xmn512M',
  '-Xms512M',
  '-Djava.library.path=temp/versions/1.14.2/1.14.2-natives',
  '-Dminecraft.launcher.brand=${launcher_name}',
  '-Dminecraft.launcher.version=${launcher_version}',
  '-cp',
  'temp/libraries/com/mojang/patchy/1.1/patchy-1.1.jar;temp/libraries/oshi-project/oshi-core/1.1/oshi-core-1.1.jar;temp/libraries/net/java/dev/jna/jna/4.4.0/jna-4.4.0.jar;temp/libraries/net/java/dev/jna/platform/3.4.0/platform-3.4.0.jar;temp/libraries/com/ibm/icu/icu4j-core-mojang/51.2/icu4j-core-mojang-51.2.jar;temp/libraries/com/mojang/javabridge/1.0.22/javabridge-1.0.22.jar;temp/libraries/net/sf/jopt-simple/jopt-simple/5.0.3/jopt-simple-5.0.3.jar;temp/libraries/io/netty/netty-all/4.1.25.Final/netty-all-4.1.25.Final.jar;temp/libraries/com/google/guava/guava/21.0/guava-21.0.jar;temp/libraries/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar;temp/libraries/commons-io/commons-io/2.5/commons-io-2.5.jar;temp/libraries/commons-codec/commons-codec/1.10/commons-codec-1.10.jar;temp/libraries/net/java/jinput/jinput/2.0.5/jinput-2.0.5.jar;temp/libraries/net/java/jutils/jutils/1.0.0/jutils-1.0.0.jar;temp/libraries/com/mojang/brigadier/1.0.17/brigadier-1.0.17.jar;temp/libraries/com/mojang/datafixerupper/2.0.24/datafixerupper-2.0.24.jar;temp/libraries/com/google/code/gson/gson/2.8.0/gson-2.8.0.jar;temp/libraries/com/mojang/authlib/1.5.25/authlib-1.5.25.jar;temp/libraries/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar;temp/libraries/org/apache/httpcomponents/httpclient/4.3.3/httpclient-4.3.3.jar;temp/libraries/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar;temp/libraries/org/apache/httpcomponents/httpcore/4.3.2/httpcore-4.3.2.jar;temp/libraries/it/unimi/dsi/fastutil/8.2.1/fastutil-8.2.1.jar;temp/libraries/org/apache/logging/log4j/log4j-api/2.8.1/log4j-api-2.8.1.jar;temp/libraries/org/apache/logging/log4j/log4j-core/2.8.1/log4j-core-2.8.1.jar;temp/libraries/com/mojang/realms/1.14.16/realms-1.14.16.jar;temp/libraries/org/lwjgl/lwjgl/3.2.1/lwjgl-3.2.1.jar;temp/libraries/org/lwjgl/lwjgl-jemalloc/3.2.1/lwjgl-jemalloc-3.2.1.jar;temp/libraries/org/lwjgl/lwjgl-openal/3.2.1/lwjgl-openal-3.2.1.jar;temp/libraries/org/lwjgl/lwjgl-opengl/3.2.1/lwjgl-opengl-3.2.1.jar;temp/libraries/org/lwjgl/lwjgl-glfw/3.2.1/lwjgl-glfw-3.2.1.jar;temp/libraries/org/lwjgl/lwjgl-stb/3.2.1/lwjgl-stb-3.2.1.jar;temp/libraries/com/mojang/text2speech/1.11.3/text2speech-1.11.3.jar;temp/libraries/org/lwjgl/lwjgl/3.2.1/lwjgl-3.2.1-natives-linux.jar;temp/libraries/org/lwjgl/lwjgl-jemalloc/3.2.1/lwjgl-jemalloc-3.2.1-natives-linux.jar;temp/libraries/org/lwjgl/lwjgl-openal/3.2.1/lwjgl-openal-3.2.1-natives-linux.jar;temp/libraries/org/lwjgl/lwjgl-opengl/3.2.1/lwjgl-opengl-3.2.1-natives-linux.jar;temp/libraries/org/lwjgl/lwjgl-glfw/3.2.1/lwjgl-glfw-3.2.1-natives-linux.jar;temp/libraries/org/lwjgl/lwjgl-stb/3.2.1/lwjgl-stb-3.2.1-natives-linux.jar;temp/libraries/com/mojang/text2speech/1.11.3/text2speech-1.11.3-natives-linux.jar;temp/versions/1.14.2/1.14.2.jar',
  'net.minecraft.client.main.Main',
  '--username',
  'Steve',
  '--version',
  '1.14.2',
  '--gameDir',
  'temp',
  '--assetsDir',
  'temp/assets',
  '--assetIndex',
  '1.14',
  '--uuid',
  'e9defc5c79564e3d83d9a90684bb05cb',
  '--accessToken',
  '824ffdfb-089a-486a-9cd7-e7250ff92846',
  '--userType',
  'legacy',
  '--versionType',
  'release'
]

Personally cannot repro this case.
Maybe there is some error happened during download & install progress.
Like corrupted version jar, the launching precondition does not check it.
I add the check in this commit
Thanks report issue anyway.
This change is included in 5.1.0

Thank you for a quick response!
I have found the problem to be the classpath separator, in retrospect I probably should have included that I am running this on Linux in the issue, as Linux also requires a colon as a path separator.
I have made a pull request #10 to fix this. Unfortunately, it is not signed.


Test round-trip serialization

In order to avoid issues like #547 popping up, we should take a closer look at our serialization and, in particular, implement round-trips from our data structures to various formats, i.e.
https://github.com/openforcefield/openforcefield/blob/af7c2f862c42ab486a3bf1ec90d36920862faba3/openforcefield/tests/test_molecule.py#L258-L259
Counter-point: How much work would it be to make each to_dict() method adhere to the various formats compared to refactoring everything to be built on top of pydantic? I don't have a good feel for how much work each of these would take, so it is hard for me to estimate. What I do know is that

we don't want bugs like this
a later refactor to pydantic or something similar seems likely, if not inevitable

So I'm not sure how to prioritize this at the moment - @j-wags @dotsdl @jthorton and others what do you think?
Related #531 and probably others

I don't have experience with pydantic beyond work on QCEngine, and though it's a bit painful it does provide some nice runtime guardrails, which can be important for core code paths. Let's continue the pydantic discussion here.
Even with pydantic, we still want tests of these serialization round trips, as you noted. I think these should go right where you pointed out, even with the #531 refactor. Do you want to take this on, or would you like me to?

I would like to take the first stab and delegate from there. I probably won't be able to do it alone

So I'm not sure how to prioritize this at the moment

Getting the tests back online and offering some working functionality to serialize conformers to JSON is pretty high priority, and we can rank exactly how high at Monday's sprint planning. But it would be great to discuss the options we have on that front if you gain some insight by looking into this sooner.
Implementing a pydantic object model will give us JSON and more, but require significantly more effort, and be a brittle design. So I agree we should consider it a separate, lower-urgency topic.


Delegating permissions to implement Trait

Currently, it is only possible to implement traits on types if either:
a. a crate owns the type
b. a crate owns the trait
The reasoning for that, as far as I understand, is that allowing random crates to implement traits on types where they own neither we quickly end up in a situation where there are competing implementations of traits in which rust can't decide anymore which is the "right one".
This can lead to an unpleasant situation for traits that are generally useful where:
A crate implementing a common, useful, widely used, the trait has to depend on and implement traits on a large number of other crates they wish to implement the trait for - usually accompanied by a number of feature flags so users can cut down the dependency graph.
A crate implementing a common type having to depend on and pull in a number of trait-crates to implement their traits. Again usually accompanied by a number of feature flags to cut down the use of unused dependencies.
An example
The probably best example for such a case is serde, which, rightfully so, a lot of crates depend on because it offers an important trait that crates might implement.
At the time of writing, there are over 5000 crates (https://crates.io/crates/serde-json/reverse_dependencies) depending on it which to a large degree, if you just use them for functionality do not require serde at all.
(note this is not a criticism on serde or any library using its traits!)
Proposal
Allowing crates to delegate the right to implement traits on their types, or implement their trait for types to other crates would allow people to opt in / out of dependencies in an independent manner from feature flags.
In other words, a crate a could specify a list of other crates (b, c, d) that are allowed to either B, C, D are allowed to implement traits in A on types that are neither owned by A nor themselves or implement traits that neither they nor A own on types owned by A.
Since permissions would need to be explicitly delegated by a create that has the permissions themselves this would not break the initial reason of conflicting trait implementations on types.
Cargo.toml (for crate a)
delegate-trait = ["a", "b", "c"] # might need a better name but that's a detail

An example (again):
The user of the url crate could delegate permissions to a crate url-serde so that url-serde could implement Serialize and Deserialize on the types defined in url. This would remove the need for future flags and people wishing to use serialization with url can explicitly add the extra dependency.
limitations
A limitation with this minimalistic approach would be that derive based traits couldn't be delegated. However, nothing in this proposal would block adding this later on with an additional construct around deriving for foreign structs.

See https://github.com/Ixrec/rust-orphan-rules for a more complete summary of why this stuff is the way it is and existing proposals haven't gained much traction.
This specific idea of the trait-owning crate allowlisting potential implementer crates usually runs into a "not solving the actual problem" objection:
https://internals.rust-lang.org/t/revisit-orphan-rules/7795/59?u=ixrec

perhaps we could brainstorm some sort of whitelisting mechanism where diesel and chrono both declare that diesel-chrono has special permission to impl their items.

That kinda defeats the purpose. The idea is that someone should be able to make this crate without either of the parent crates having to know/care/get involved

@lxrec was faster. But here is an example that illustrates why this would be hard to solve.
I think this could work if you only delegate permissions to a single crate. But this kind of defeats the purpose. An IMO you can just as easily use feature flags for this (and might even be more flexible with feature flags).
If you allow multiple other crates to implement: What happens if there are actually multiple crates that do implement the trait for your types somewhere in your dependency hierarchy? For example because you have both b and c as a dependency.
This issue could probably be avoided for the direct dependencies of your crate, but it gets more complicated (or impossible) for indirect dependencies. E.g. for something like this dependency tree:
YourCrate
- Def1 (delegates permission to crates ImplA and ImplB)
- SomeDep1
  - Def1
  - ImplA
- SomeDep2
  - Def1
  - ImplB

In this scenario you wouldn't be able to use both SomeDep1 and SomeDep2 together because ImplA and ImplB aren't be allowed to be used together.

This issue could probably be avoided for the direct dependencies of your crate, but it gets more complicated (or impossible) for indirect dependencies. E.g. for something like this dependency tree:

The exact same issue exists today with feature flagged implementation of traits. If you have a the-type create and a the-trait crate where that feature flag each other for implementations of the trait and you have two indirect dependants that include the respective feature flage.
I would go so far that this is "worse" then what would be introduced with this proposal as the implementer of the-type and the-trait do not ever need to have communicated while the owner of Def1 and ImplA and ImplB will need to have communicated (and I suspect but that's just a suspicion) will mostly be the same person.

I'd think the serde case should always be handled by optional dependencies, but optional dependencies could be improved dramatically.  We could make it far easier use dependencies only when another crate requires them.
[dependencies]
serde = { version = 1.0^, optional = implicit }

I think delegation proposals envision crate authors creating micro subcrates, but doing this seems far less user friendly than the current optional dependencies scheme, not only for the crate developer, but also downstream.  In short, the serde example gives a reason why rust should not do this even if it could do so.

Note that delegation and optional dependencies/automatic features are targeting different kinds of orphan rule pain. AFAIK there's a rough consensus that we probably would like both in some form, but finalizing either of them is not a priority for now.
(and as usual, we're just recapping points made a million times in previous threads about this)

I think delegation proposals envision crate authors creating micro subcrates, but doing this seems far less user friendly than the current optional dependencies scheme, not only for the crate developer but also downstream. In short, the serde example gives a reason why rust should not do this even if it could do so.

I'm curious how you come to the assessment that it's less user friendly. feature flags are entirely opaque, often the only way to find them is to inspect the Cargo.toml of a crate. Removing default features is problematic, you got to remove all and then add the ones back that you want then hope that in the future the defaults don't change and break things. Removing partial flags that are pulled in by implication is also not possible.
On the other hand, dependencies are a much more elaborate system with more capabilities, they're easily explorable, they're (mostly) independent of each other by design, they allow overwriting via the cargo toml if needed.
There is an additional burden on releasing more then one new version if a trait fundamentally changes to the level of breaking backwards compatibility.
@Ixrec I might be missing something but the discussion you linked was about allowing "random" traits to implement Features on Types, there is a very distinct difference in doing that and allowing explicit delegation of permissions. It's erveryone-has-root vs. there-is-a-sudoer-group. But I probably might miss other discussions on the topic?

The repo and the IRLO thread I linked both cover a lot of ground (especially if you follow all of the other stuff they link to), including forms of allowlisting as proposing here, abandoning the orphan rules to allow "random" impls, and just about everything else in between.

I'm curious how you come to the assessment that it's less user friendly. feature flags are entirely opaque, often the only way to find them is to inspect the Cargo.toml of a crate. Removing default features is problematic, you got to remove all and then add the ones back that you want then hope that in the future the defaults don't change and break things. Removing partial flags that are pulled in by implication is also not possible.

Given that you appear to have opened this in response to a tweet about the discoverability and related UX elements of features being terrible, I think it'd be more productive to discuss the root problem there, rather than just taking for granted that this is the best solution.


[QUESTION] ShortCut Keys aren't working (CMD+Shft+A Q) and git add all commit push (CMD+shft+A P)

[QUESTION]
ShortCut Keys for Git add commit push (CMD+Shft+A Q) and git add all commit push (CMD+shft+A P)
Steps to reproduce:
No worky:

Fresh install of ATOM + git-plus being the only additional package

Worky:

Installing the now deprecated git-plus-plus actually adds the hotkey functionality

the changelog has a note about that change https://github.com/akonwi/git-plus/blob/master/CHANGELOG.md

Thanks @akonwi


Cannot find module 'nativescript-angular/forms'.

What version of angular2 are you using?  More errors......  I mean in  the tutorial it said to add to app.modules.ts but it was not there so I had to create.  Is this tutorial up to date with angular2 official release?
app/app.modules.ts(1,10): error TS2305: Module '"/Users/davidmontgomery/Documents/frontend/sample-Groceries/node_modules/@angular/core/index"' has no exported member 'NgModule'.
app/app.modules.ts(2,41): error TS2307: Cannot find module 'nativescript-angular/forms'.
app/app.modules.ts(3,36): error TS2307: Cannot find module 'nativescript-angular/platform'.

Hey @davidmontgom,
Sorry about this. There was actually a small error in the tutorial. The chapter should’ve said app.module.ts and not app.modules.ts. I’m patching that up now.
nativescript-angular/forms is included as part of the 1.0.0 nativescript-angular package from npm, which should be included when you clone the Groceries repository. Can you let me know if you’re still having issues here?
Thanks!

Closing due to inactivity. @davidmontgom if you’re still having issues let me know.


Optimistic local mutations without setting dedupingInterval to 0

Hi! I was looking for alternatives to SWR in Vue and I found this library, which is great!
I found how to do optimistic local mutations in this issue (#83), but I'm kind of worried about having to set dedupingInterval to 0, because it means we can't really use deduping intervals and optimistic mutations at the same time.
Looking through the code, I think putting these lines inside an if could work.
Is there a way to do this I haven't found? Otherwise, would you be willing to accept a PR to add this functionality?

@davidblnc sounds good, we should add the shouldRevalidate argument to mutate with the default to true, then you can send false without setting dedupingInterval to 0
https://github.com/vercel/swr/blob/master/src/types.ts#L68-L72
https://github.com/vercel/swr/blob/master/src/use-swr.ts#L102

Unfortunately I haven't been able to get the tests to pass.
I attach the test code in case someone can give a few pointers towards the solution.
it('mutate respects "shouldRevalidate" option', async done => {
  const loadData = key => new Promise(res => setTimeout(() => res(key), 100))
  let firstFetch = true;
  const vm = new Vue(defineComponent({
    template: `<div>hello, {{data}}</div>`,
    setup() {
      const { data, mutate: revalidate } = useSWRV("mutate-should-not-revalidate", () => {
        if (firstFetch) {
          firstFetch = false
          return 'initial-state'
        }
        return 'revalidated-state'
      });

      setTimeout(() => {
        mutate("mutate-should-not-revalidate", loadData('optimistic-update-state'), undefined, undefined, false);
      }, 100);

      setTimeout(() => {
        revalidate()
      }, 300);

      return { data };
    }
  })).$mount();

  expect(vm.$el.textContent).toBe('hello, initial-state')

  timeout(200)
  await tick(2)

  expect(vm.$el.textContent).toBe('hello, optimistic-update-state')

  timeout(100)
  await tick(2)

  expect(vm.$el.textContent).toBe('hello, revalidated-state')

  done()
})

@davidblnc this test passes for me on master (since i don't have any code that has shouldRevalidate argument). It should fail once you implement the shouldRevalidate functionality and revalidated-state should not happen. Maybe open a pull-request and we can chat there. Also, I think maybe we should move the mutate signature to be
type MutatorOptions = {
  shouldRevalidate: boolean
  cache: SWRVCache<Omit<IResponse, 'mutate'>>
  ttl: number
}
export type Mutator<Data = any> = (
  key: IKey,
  data?: Data | Promise<Data>,
  options?: MutatorOptions
) => Promise<Data | undefined>

const mutate: Mutator = async (key, res, options) => {
  // ...
}

// e.g.
mutate("mutate-should-not-revalidate", loadData('optimistic-update-state'), {
  shouldRevalidate: false
})

which will give us the ability to fallback to default cache and ttls.

After testing a bit, I think the library actually doesn't revalidate after calling mutate.
In this example, I would've expected to see:

First the initial state (initial)
Then the optimistic state (optimistic)
Finally, mutate calls the fetcher again to revalidate and it prints "initial"

@davidblnc whoops you're right... the mutate actually needs to be added as a feature https://github.com/Kong/swrv/issues/92#issue-716972418

I'm trying to figure out how I can get a refetch/revalidate while in the mutate function.


Sub-regions, repeated data and .geojson hosting

To be honest, I'm not sure where would be best to ask about this, either here, location-conflation or the Name Suggestion Index, so apologies if this isn't relevant here.
Basically, over on the Name Suggestion Index, there is a /features/ folder that I've expanded recently, and it contains .geojson files for mapping certain areas that a brand should be limited to, and this is powered by location-conflation and seems to work well.
However, I've noticed that the osm-community-index has the same concept, with a /features/ folder linking areas where an OSM community group is based.
Within both projects are similar region files, and I can see both projects expanding with similar data. One example being the West Midlands in England (NSI vs OCI) which is basically the same region.
I imagine other projects exist (or will exist) with the same goal of having sub-regions where the .geojson files are similar.
I gather that this project (country-coder) doesn't intend to expand into sub-regions, so as to prevent it becoming too large, and I can understand that.
However, is there any scope for hosting sub-region .geojson files within the repository (or location-conflation) so there is a central place for users to reference the region files, without having to recreate them for their own project?

We have the following projects which work this way now:

osm-community-index (probably has about as many custom geojsons as it needs)
name-suggestion-index (potential for many custom geojsons)
id-tagging-schema (needs few if any custom geojsons)
and then there's background imagery 🤣  the less we say about that, the better.

I'm ok with just leaving the features/ folders in the separate projects for right now, without trying to establish a central place for them.  Each project should just bring the minimal amount of geojsons that it needs.  The goal is to ship less stuff with core iD, then pull it in only as needed.  I'm not too worried about duplicating kilobytes of geojson.  iD is already pretty data heavy, but most of that cost comes from downloading geodata from OSM and imagery tiles from wherever.
We also use a few tricks on the iD side to work with data when the browser is idle.  For example, the community index gets downloaded and setup in the background while the user is either preparing their changeset comment or waiting for the upload to finish - so even if it got bigger or had redundant data in it, nobody would notice.


template function safety

Mithril framework say that there is a security hole in underscore template function. Test results are at there home page. What you gays think?

This could be avoided by using <%- val -> instead of <%= val %>
http://jsbin.com/nicivikife

Saying underscore templates are unsafe by default is ingenuous. Use <%- %> instead of <%= %>.

Apologies, I didn't mean to imply it's you, @oshanz. It's Mithril, and it's contributors should know better.

You can ping @lhorie about it.

Uh, no, I'm not claiming there's a security hole in Underscore templates. What that demo aims to demonstrate is that it's easy for a developer to unintentionally do the wrong thing.


Suggestion: Ability to filter by mob lvl

Loving the plugin so far! is there any way to have an option to hide/filter mobs below a certain threshold and or non aggro mobs? Like if i'm running around at lvl 90 and going through an area that is lvl 1-5 mobs I don't really need to see them. An option to hide mob tags say below 10 (or adjustable) lvls of character lvl would be amazing!
If it would be possible to have different color options for Hunt board mobs as well that would be amazing.
Thanks

I plan to work on the level below you check soon. Should not require too much new.
As for hunts, I don't have any list of dataids for hunt mobs. Would have to make that or find it somewhere. If you have a list of the hunts I could start separating them the way deep dungeon mobs are separated already.

That's great to hear about level below, i'll definitely see what i can do about getting a list together. One thing i did notice, not sure if it's stock or if there's a plugin i have doing this (I tried disabling a couple i thought might do this but nada) but i did see there's a special icon on their head signifying it's a hunt mob. Not sure if there's any way to read that through the plugin so it "creates" the list iteself.

 No clue

The level-based rendering has been completed. There is now an option to have a separate config for mobs some levels below you. I had a few ideas for a more in-depth implementation but didn't want to make it too complex atm. This does not work for eureka or deep dungeons as of now.
Hunts still pending more investigation. It doesn't look like the mobs have any special flags, and other plugins (specifically HuntHelper) does it by a manually created list of hunts. Could simply copy it over, but still looking into how that can best be done.

Awesome, yeah i was going to suggest to look at HuntHelper as i was using that as well. I think your plugin if able to mark the mobs different along with Helper would be a great combo.

#28


Feature Request: Ability to re-fetch a specific pool's metadata using ID

The issue originates as a requirement on Daedalus end, but since pool queries are made against cardano-wallet , I believe the issue belongs here.
Allow users who already know a pool ID they want to delegate to, to be able to refresh/download the metadata independently of SMASH server status.
Currently due to various design complications/networking reasons, a pool's metadata may be fetched and not be refreshed for a while (mostly due to the way SMASH works, but also we save local pool metadata). This means at times delegators have to wait for days if not hours for a correctly registered pool metadata.
The feature request is to allow a specific pool ID to be refreshed directly from wallet layer - without having to wait indefinitely upon when SMASH server will correct the information (while we can elect a different smash server from daedalus, there will not be a way to add any community instances to pre-populated fields in UI layer as part of default installation, and users will not be bothered looking around for custom smash server).

Hi, what's your exact problem here?
Do you have a newly registered pool, or an existing pool for which you would like to change the metadata?

@rvl - I do not have a problem myself, but across community channels there are 100s of cases where pool operators who either update metadata or register new pool - at times (due to limitations on SMASH now, before used to be wait times on wallet) often end up getting frustrated and anxious because the updates are not realtime. In fact it is becoming a common caution to not touch the pool metadata knowing the delays/inconsistencies with the Daedalus -> Wallet -> SMASH architecture.
Hence, I phrased it as a feature request (as against any bug - since things are working as they were developed).

If an operator updates anything on their existing pool meta, even if everything is correct metadata hash is correct, no url conflict or cache issues, it can still be hours or days for the wallet to update and sync. This new update leaves pools left in the "unknown" status and missing from the wallet UI for sometimes days it appears Daedalus has cached the old information and is conflicting with the new update which leaves the pool in extended unknown state. We've had several reports of this issue and they are increasing.
Instead of triggering a full refresh on metadata which increases load and takes close to an hour to complete currently. I believe the solution / feature being requested is a direct refresh of the db  for a specific known pool at wallet layer itself (ex: for Daedalus users, additional button next to search to refresh pool info using metadata URL instead of SMASH) vs waiting on smash to update currently (which can be unknown ETA), or trigger a refresh of pool DB (which could take hours).

The cardano-wallet release this week will contain PR #2432 which will make refreshing all metadata faster.
If pool operators need to quickly check that their metadata looks correct then they could disable SMASH.
I do not know why the SMASH server introduces long delays.
If not otherwise busy, the wallet should fetch new metadata fairly promptly after it finds a registration certificate with a new metadata hash.


CLI arg to select specific mutation types

Potential mutations are getting large - adding a CLI arg to opt-in or opt-out of particular categories of mutations could be useful.

wlbl branch is starting this - added --whitelist and --blacklist with two-letter category codes for each major mutation set type.
Need to restrict sample space of potential locations based on set decisions. Markers with TODO in controller on likely restriction location.

Added core function and tests to dev, remaining TODO is to provide write-up example in readme and release.

https://github.com/EvanKepner/mutatest/releases/tag/0.9.1


Update github.com/minio/minio-go

Closes #996

Codecov Report

Merging #1010 into master will decrease coverage by 0.42%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1010      +/-   ##
==========================================
- Coverage   53.91%   53.49%   -0.43%
==========================================
  Files         116      117       +1
  Lines       11000    11101     +101
==========================================
+ Hits         5931     5938       +7
- Misses       4391     4485      +94
  Partials      678      678

Impacted Files
Coverage Δ

restic/backend/layout_rest.go
73.91% <0%> (-15.57%)
:arrow_down:

restic/backend/layout_default.go
75% <0%> (-15%)
:arrow_down:

restic/backend/layout_s3legacy.go
81.81% <0%> (-11.29%)
:arrow_down:

restic/backend/s3/s3.go
66.31% <0%> (-4.1%)
:arrow_down:

restic/backend/layout.go
71.59% <0%> (ø)
:arrow_up:

restic/fuse/dir.go
0% <0%> (ø)
:arrow_up:

cmds/restic/cmd_migrate.go
3.7% <0%> (ø)

restic/fuse/file.go
68.1% <0%> (+0.55%)
:arrow_up:

restic/fuse/snapshot.go
3.73% <0%> (+3.73%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1aab123...a325a20. Read the comment docs.


Changes to java-spring-boot2 stack to facilitate downstream selection of snowdrop

Checklist:

[X] Read the Code of Conduct and Contributing Guidelines.

[X] Followed the commit message guidelines.

[X] Stack adheres to Appsody stack structure.

Modifying an existing stack:

[X] Updated the stack version in stack.yaml

Contributing a new stack:

Describe how application dependencies are managed:

Explain how Appsody file watcher is utilized:

Describe other Appsody environment variables defined by the stack image:

Describe any limitations and known issues:

Related Issues:

@neeraj-laad Please note: ^^^

@neeraj-laad I see a check remains outstanding:

license/cla Expected — Waiting for status to be reported

However it is noted above:

All committers have signed the CLA.

Please advise on what I need to do next.


#2095 add   skipnontestassemblies funcionality to nunit3 runner

What You Are Seeing?
NUnit3Settings class does not have property SkipNonTestAssemblies
What is Expected?
NUnit3Settings class does have property SkipNonTestAssemblies and features behind it as described here: https://github.com/nunit/docs/wiki/Console-Command-Line
What version of Cake are you using?
v0.26.1
Are you running on a 32 or 64 bit system?
64
What environment are you running on?  Windows? Linux? Mac?
Windows
Are you running on a CI Server?  If so, which one?
NO

Excause me. Could some please explain me why this CodeFactor system rejects my pull request?
After the first commit (https://github.com/cake-build/cake/pull/2100/commits/b5fdf60c49f66348768d7deeb40068b72fcaa663) it stated that I've made some code duplication in lines of code I didn't change. Despite that I've decided to fix this duplication, but CodeFactor seems to be unware of my changes and still rejects this pull request pointing the same lines of duplicated code.
Do I get something wrong?

@pavzaj your changes have been merged, thanks for your contribution 👍

Nice. Thank you.


fix(swap): Fix asset selection list bugs

related:

https://emurgo.atlassian.net/browse/YOMO-760
https://emurgo.atlassian.net/browse/YOMO-761

@jorbuedo when accessing the token list I still can see a spinner https://recordit.co/ZDwyozO2p7 and the skeleton loader after but for a very short time 🤔

Yes, I didn't change the main spinner, wasn't sure if that was part of the design or not. That one retrieves the list, then the individual ones each get the tokenInfo use the skeleton design.
Maybe I could substitute the spinner for a fixed array of skeletons

@jorbuedo when accessing the token list I still can see a spinner https://recordit.co/ZDwyozO2p7 and the skeleton loader after but for a very short time 🤔

Yes, I didn't change the main spinner, wasn't sure if that was part of the design or not. That one retrieves the list, then the individual ones each get the tokenInfo use the skeleton design. Maybe I could substitute the spinner for a fixed array of skeletons

@jorbuedo  can you check with Rahul how the expected behaviour will be. I think they expect to see only the skeleton loading there but not 100% sure of it

@jorbuedo when accessing the token list I still can see a spinner https://recordit.co/ZDwyozO2p7 and the skeleton loader after but for a very short time 🤔

Yes, I didn't change the main spinner, wasn't sure if that was part of the design or not. That one retrieves the list, then the individual ones each get the tokenInfo use the skeleton design. Maybe I could substitute the spinner for a fixed array of skeletons

@jorbuedo can you check with Rahul how the expected behaviour will be. I think they expect to see only the skeleton loading there but not 100% sure of it

I agree it should be just skeletons, it just can't be the same skeletons. At least how it's structured now. There's 2 loading boundries, one for the initial load, then for each entry. I've put a list of skeletons now. But the header of the list doesn't show until the list is loaded. I don't think it's too much of a problem.

@jorbuedo when accessing the token list I still can see a spinner https://recordit.co/ZDwyozO2p7 and the skeleton loader after but for a very short time 🤔

Yes, I didn't change the main spinner, wasn't sure if that was part of the design or not. That one retrieves the list, then the individual ones each get the tokenInfo use the skeleton design. Maybe I could substitute the spinner for a fixed array of skeletons

@jorbuedo can you check with Rahul how the expected behaviour will be. I think they expect to see only the skeleton loading there but not 100% sure of it

I agree it should be just skeletons, it just can't be the same skeletons. At least how it's structured now. There's 2 loading boundries, one for the initial load, then for each entry. I've put a list of skeletons now. But the header of the list doesn't show until the list is loaded. I don't think it's too much of a problem.

It's simpler to reproduce it on the stories. by injecting timed api responses, design the screen state there, so we can present to QA/Product, and ask for confirmation

It's simpler to reproduce it on the stories. by injecting timed api responses, design the screen state there, so we can present to QA/Product, and ask for confirmation

It is there, I've added fetching state to both. I'm trying to figure out why the sell one doesn't work though, something with the suspense I think. The buy one shows it correctly

It's simpler to reproduce it on the stories. by injecting timed api responses, design the screen state there, so we can present to QA/Product, and ask for confirmation

It is there, I've added fetching state to both. I'm trying to figure out why the sell one doesn't work though, something with the suspense I think. The buy one shows it correctly

The Boundary component is adding the loading spinner as well when fetching - maybe got something to do with it as well

It's simpler to reproduce it on the stories. by injecting timed api responses, design the screen state there, so we can present to QA/Product, and ask for confirmation

It is there, I've added fetching state to both. I'm trying to figure out why the sell one doesn't work though, something with the suspense I think. The buy one shows it correctly

The Boundary component is adding the loading spinner as well when fetching - maybe got something to do with it as well

I'd suggest to download Flipper, it allows to see the react tree, then you can toggle suspense on any component and it will show you which of the Boundary is kicking it.

It's simpler to reproduce it on the stories. by injecting timed api responses, design the screen state there, so we can present to QA/Product, and ask for confirmation

It is there, I've added fetching state to both. I'm trying to figure out why the sell one doesn't work though, something with the suspense I think. The buy one shows it correctly (more or less). Looking into it

I remember @SorinC6 mocked some responses but dunno.

@SorinC6 @stackchain Well they work differently because one has usePairListByToken and the other doesn't. There's a parade of hooks calling each other, but only the hooks that actually use useQuery directly, and have suspense to true, will suspend.
Both suspend for the individual elements of the list, but on the parent level, only buy suspends because of the usePairListByToken . sell is briefly on a state of empty list instead, while the parade of hooks figure out that the list is indeed not empty.
I think current state is stable enough to merge. Even if it has some quirk, doesn't really look wrong. To have proper bondaries working means refactoring those hooks.


Support for Portenta H7

Describe the feature
Higher end devices like the Portenta H7 do not provide pgmspace.h. This leads to a compilation error when trying to build code which uses mWebSockets on such a device (file not found error). One way to solve this issue is to provide a pgmspace.h file on the device driver side (more info about this here: https://github.com/arduino/ArduinoCore-mbed/issues/343). Another way is to implement device conditional compilation, meaning adding Portenta H7 to here and here.
@skaarj1989 @facchinm I could provide a PR if you are ok with the second alternative how to extend support of the lib. If not one would have to take the first approach instead. It seems like there is no best practice about which approach to prefer over the other in the Arduino ecosystem BTW.
Environment info
n.a.
Expected behavior
Should compile on Portenta H7.
Additional context
n.a.

@fkromer If you have a Portenta board, could you verify if the code is working?

@skaarj1989 Mh, the example will not work via Wifi I guess. The Wifi chip on the Portenta H7 is LBEE5KL1DX. You are expecting some Wxxxx chip, right?

@fkromer Server running at <IP_ADDRESS>:3000 Obviously this is bad.
Try to run WebClient demo and see if you can get valid IP address.

Obviously this is bad.

@skaarj1989 Yepp

@skaarj1989 I've tried with a router and static IP. The server should be running on the static IP.
Initializing ...
Server running at <IP_ADDRESS>:3000

However I cannot establish a websocket connection. I get an error: socket hang up.

@skaarj1989 Just want let you know that I do not need this feature anymore.

Anyways, support for Portenta board would be nice to have.


COREE-1952/ci

Make adjustments to Makefile for using in CI. Make minor adjustments to existing adoc to get compile/lint steps reasonably passing. See inline comments for change explanations.

Update the readme with local instructions. You can still use docker. Just mount the local path and use the asciidoc image, replacing the entry point with the path to make... assuming it had make installed.

Update the readme with local instructions. You can still use docker. Just mount the local path and use the asciidoc image, replacing the entry point with the path to make... assuming it had make installed.

As discussed offline, this is really about directions to run locally with make as the entrypoint. As for the documentation, I was going to rewrite the README as a separate PR. Given you are out now and I'd like you to review that, I will do it in a separate PR.


SidebarCollapsibleCard: Toggle open with prop change

SidebarCollapsibleCard: Toggle open with prop change
This update allows for the SidebarCollapsibleCard open state to be influenced by the prop change of isOpen, allowing for more programmatic control from a parent component/container.

Coverage remained the same at 100.0% when pulling ad14c2c880574508dc804adf69ac0537b7a31a9b on sidebar-collapsible-card-toggle-open-with-props into 97a607b3cb15bb0878236aa2bd449855eda95047 on master.


Graham Scan sorts already sorted points

For the merge step in Divide and Conquer, the points are already sorted when they are passed to the Graham Scan algorithm.
The functionality of sorting and convex hull construction should be split.

Made sorting for Graham scan optional, removed the double sort for Divide and Conquer join step.


is this code still works?

I've pulled this code to crawl some specific users mentions, but can't get through more than ./bootstrap.sh
I'm wondering because it only generates empty folders.

Yes, the code still works.  A few others have been successfully running this.
How was  your setup?


Slate cannot compile Bootstrap 4

Problem
I have tried to get Bootstrap 4 Alpha 6 SCSS to work with slate, however, when I try typing 'slate start' in the terminal, the files are successfully synced to the Shopify store. But in the inspect console, the file says 'can not get' and if you open the file in a new tab it says: 'Failed to compile SCSS file'.
Does this have anything to do with the version of Sass Slate uses (A fork of 3.2)?
Sass Version 3.3
In Sass version 3.3 they introduced Sass script maps which seems to be one thing that breaks the compilation of Bootstrap:
http://sass-lang.com/documentation/file.SASS_CHANGELOG.html#selector_functions

This would be more of a general Shopify issue.  Currently, Shopify's CDN servers that compile the SCSS files support Sass 3.2.6.  So technically, Bootstrap 4's SCSS files that require at least 3.3 will not compile in any Shopify theme.

Thanks @jonathanmoore
@mblode I took a look at the repo for bootstrap-sass and see that Sass version 3.3.4 is required.  https://github.com/twbs/bootstrap-sass/blob/master/bootstrap-sass.gemspec#L14
You'll need to compile your Sass separately as Shopify's compiler won't be able to do this.

Yeah, I'm compiling bootstrap 4 with my own custom variables using Codekit and then just importing it. Is there any goal to update Sass from the 5-year-old version Shopify is currently using?

Is there any goal to update Sass from the 5-year-old version Shopify is currently using?

It's an issue we're aware of, but no timeline I can offer.

Is there any way to update the 'Failed to compile SCSS file' to reflect the actual error. I'm finding it near impossible to convert my site to being compiled by shopify

Yes, it would be so nice and much easier to have Sass updated or at least have an option to choose which version to use so it doesn't break old themes. Ending up using compiled version of Bootstrap 4 Grid system (by CodeKit) and imported to the current theme.


 I didn't see any contribution guidelines, I hope you are okay with such a small PR :)

Thanks, Simon!
v2.1.1 just has been released 🎉
And yes, I don't have any contribution guidelines yet. The library is small and simple, so I want to keep pull requests "free" and as simple as possible so as not to discourage contributors :)


add how to generate bearer tokens

Why:
This intro guides introduces a token, but does not explain what it is or where to get one (I needed to click on the next guide). This makes the intro guide somewhat more self-contained in that it can be completed without needing to begin the next guide, i.e. allow the follower to get a nice pleasant "yay it worked" feeling, though keeping it optional with a note as details of course follow in the next guide.

What's being changed:

Improving the introduction-to-graphql.md guide by adding the error message and link to where tokens come from.
Check off the following:

[x] I have reviewed my changes in staging (look for the latest deployment event in your pull request's timeline, then click View deployment).
[x] For content changes, I have completed the self-review checklist. (Whoa... ok done to the best of my ability - that's a lot of rules and not much grammar bot to auto-enforce it!)

Writer impact (This section is for GitHub staff members only):

[ ] This pull request impacts the contribution experience

[ ] I have added the 'writer impact' label
[ ] I have added a description and/or a video demo of the changes below (e.g. a "before and after video")

@pzrq
Thanks so much for opening a PR! I'll get this triaged for review :zap: