convert frwj report for wmarchive

remove the layer, ("steps",  "checksum", "dataset", "sources")
convert name in output =  {"OutputPFN": "outputPFN", "inputPath": "inputDataset",
"lfn": "outputLFN", "input": "inputLFNs", "inputpfns": "inputPFNs"}
remove the field from output  ["InputPFN", "pfn"] - not useful information
convert the dataset pattern {"outputDataset": ["primaryDataset", "processedDataset", "dataTier"]}
change the run structure  {'runNumber': 123, lumis: [123, 124]}

The example is updated https://github.com/dmwm/WMArchive/wiki/fwjr.json
@yuyiguo, @vkuznet, please review.
There still need to be set the default values correctly (please point if you find any problems)

@ticoann are you trying to shrink this data by removing only outer dict level? Or could it be a few inner keys only?
I believe data volume will be critical here, so we should get rid of any useless and repeated information, right?

Seangchan,
the schema is improving, but still has many holes in it.
Basically all empty dicts or lists should be filled with whatever structure
they may contain. Here are they are:
{'cmsRun1': {'analysis': {},
             'cleanup': {},
             'errors': [],

....

'fallbackFiles': [],
 'logArch1': {'analysis': {},
              'cleanup': {},
              'errors': [],
              'input': {},
              'logs': {},
			  'output':[...
				'runs': [],
				...
			  ],
			  'parameters': {},
              'performance': {'cpu': {},
                              'memory': {},
                              'multicore': {},
                              'storage': {}},

'skippedFiles': [],
 'stageOut1': {'analysis': {},
               'cleanup': {},
               'errors': [],
               'input': {},
               'logs': {},
               'output': [],
               'parameters': {},
               'performance': {'cpu': {},
                               'memory': {},
                               'multicore': {},
                               'storage': {}},
....

But for some attributes, e.g. performance dict, I don't think
I understand why cpu/memory/multicore/storage should be a dict types?
Best,
Valentin.
On  0, ticoann<EMAIL_ADDRESS>wrote:

The example is updated https://github.com/dmwm/WMArchive/wiki/fwjr.json
@yuyiguo, @vkuznet, please review.
There still need to be set the default values correctly (please point if you find any problems)

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-187427883

@ticoann are you trying to shrink this data by removing only outer dict level? Or could it be a few inner keys only?

As far as i understood, nested structure give more problem on the data structure (searching) used by hadoop. Valentin would know better about this.

I believe data volume will be critical here, so we should get rid of any useless and repeated information, right?
I'm not sure how it would be critical with data volume. But yes, we will try to reduce the duplicate information as much as possible.  We talked Dirk briefly some of the data would be exactly the same but has different meaning depend on the section it resides.
i.e. run and lumi mask in input section has different meaning than the same info in output section.
Input is what we try to intend and output is what we get. It is the since the unit is a job. But it could be different when we break the job in smaller unit.

But mostly I am sure what all the information is in framework job report how this will evolve. So please advice if you find any information which could be removed.

Hi Valentin, I updated information with some fake data in  https://github.com/dmwm/WMArchive/wiki/fwjr.json.
Basically all the sections originally in steps has the same structure.
So cmsRun1, logArch1, stageOut1 will have the same structure. (Although some ot them are always empty in logArch1 and stageOut1)
For performance data it is configurable so key values are arbitrary (but with in the list of the choices)
I am not sure what are the list of choices for each group. But the example is what is set in agent currently - which never changed. Although it could change, but I am not sure what would be added. So we have to change the schema when that happens.
For
'analysis': {},
'cleanup': {},
'logs': {},
'parameters': {}

I am not sure about the structure of those field.  As far as I checked, I couldn't fine any data there. So I am not completely sure what the structure would be. We can completely remove those but not sure whether that field will be field up some times.  My guess is analysis part would be used in crab, but not completely sure.
Thanks

test this please

My comment will completely depend on whether we prefer saving disk space or cpu cycles. In case we want to save space, here are my comments (based on the json example):

under step / errors, we have a list of errors. Do you know whether we are aggregating that info? Or if we have 10 errors exactly the same, that list length will be 10?
under step / input, 'guid' is already party of the lfn, so it could be removed (unless we want to use it to optimize database lookup).
under step / output, maybe we could also remove 'async_dest', 'catalog', 'guid' and 'cksum' (aren't we using only adler32 now?)? Still here, I'm not convinced we should keep 'user_dn', 'user_vogroup' and 'user_vorole'.

Alan and others,
since we're talking about long term storage and backed up by HDFS we don't need
to worry about size to some extend. If you add more fields that's fine. We also
PLAN to do aggregation therefore anything useful from terms of aggregated
statistics should go to the report. If you need to find who generated samples
then user DN we must have. Basically, your questions should be converted to
what you want to do with stored document. Any queries you foresee must have
appropriate information in FWJR.
But as Seangchan mentioned we should try to avoid nested structure, i.e.
dict contains list of dicts, which by itself contains other dict/lists.
This structure is easy when you construct the document by really hard to use in
searching. The flatten structure is better for storage and search.
Ideally I would prefer to have one level of nestness, e.g. a dict with
key/value pairs, but not a dict with another dict as a value for some keys.
Such flatten structure is easy to extend and that is allowed, e.g. we can
expand schema, but we should not shrink it.
V.
On  0, Alan Malta Rodrigues<EMAIL_ADDRESS>wrote:

My comment will completely depend on whether we prefer saving disk space or cpu cycles. In case we want to save space, here are my comments (based on the json example):

under step / errors, we have a list of errors. Do you know whether we are aggregating that info? Or if we have 10 errors exactly the same, that list length will be 10?
under step / input, 'guid' is already party of the lfn, so it could be removed (unless we want to use it to optimize database lookup).
under step / output, maybe we could also remove 'async_dest', 'catalog', 'guid' and 'cksum' (aren't we using only adler32 now?)? Still here, I'm not convinced we should keep 'user_dn', 'user_vogroup' and 'user_vorole'.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-187574705

Seangchan,
even though I understand as human your statement

Basically all the sections originally in steps has the same structure.
So cmsRun1, logArch1, stageOut1 will have the same structure. (Although some ot them are always empty in logArch1 and stageOut1)

we MUST fill out those structure with some fake values for the schema file.
Please do so. You need to understand that once we put schema in code it will
check it at RUN TIME and if schema file has empty data-structure and at RUN
TIME it will be provided with "the same structure" the code will just BREAK (or
worse seg-fault/throw exceptions).  Don't think in python way, think in terms
of C/C++ code. You can't have undefined variable or change data type of the
variable in C/C++ code.
So, fill out those "identical" structures in cmsRun1/logArch1/stageOut1.

For performance data it is configurable so key values are arbitrary (but with in the list of the choices)
I am not sure what are the list of choices for each group. But the example is what is set in agent currently - which never changed. Although it could change, but I am not sure what would be added. So we have to change the schema when that happens.

I rather prefer to change code that you WILL NOT assign arbitrary stuff. The
schema change is a consequence of code structure and it is better to let code behave
correctly. Unfortunately this is "bad behavior of python" which allows to assign
at run-time anything and change data-type of variables at run-time.
Think about this way. We'll be running in auto mode where each agent will feed
FWJR into WMArchive. If one agent decided to add some fields on a fly at
run-time it means ALL docs will be failed during injection into WMArchive. Can
you imaging how much time we will spend cleaning up such mess, verifying which
docs were correctly written and which are not. To avoid that it is much better
to only allow agents to change FWJR with scheduled upgrade. Before the software
upgrade we change schema and up-load it and then we change agents.
Please, please don't put us into the mess because of these lazy statements.
Change the code to make it predictable.

For
'analysis': {},
'cleanup': {},
'logs': {},
'parameters': {}

I am not sure about the structure of those field.  As far as I checked, I couldn't fine any data there. So I am not completely sure what the structure would be. We can completely remove those but not sure whether that field will be field up some times.  My guess is analysis part would be used in crab, but not completely sure.

Hmm, but there is a code which generates at least keys and associate with them
empty dicst, right? Where this code? We MUST find it. Don't remove
data-structures, remove the code which generates them. If they will be required
then we'll need a code which will generate them properly. Here again, the
change of schema will be a consequence of agent code change which will bring
these data-structures in controlled way. For example, new release will create
them and before deployment of new release we'll be able to generate new schema
with them and put it in place before we'll acquire the data.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-187516625

Seangchan,
here is more stuff to fix:

fix data-types, e.g.
"PeakkValueRss": "0", should be "PeakValueRss": 0.0
"PeakValueVsize": "0" should be "PeakValueVsize": 0.0
"TotalJobCPU": "0.39894", should be "TotalJobCPU": 0.39894
....

identify correct data type, e.g.
'size': 0, should it be int or float?

'location': None should be 'location: "some string"
'user_dn': None, should be 'user_dn': "some string"
etc.

My argument for removing the user DNs is that it was made available and - I think - never used. Probably with the assumption that it would be eventually used. Since it has passed a few years without using it, I think it'd be good to get rid of it.
BTW, there is no single person responsible/owner of the datasets, it all comes through McM with the same user/cert. The Physics group is encoded in the request names.

@ticoann , @vkuznet
Shengchan, can you merge this PR? Also, can you copy recent  fwjr to vocms013 that has bunch of free storage?
Valentin,
couchDB seems not to include in the software sets on vocms013. Instructions on installing couchDB cmsweb based node?
Thanks,
Yuyi

Yuyi, why do we need couch on a node where we're not going to use it?
On  0, Yuyi Guo<EMAIL_ADDRESS>wrote:

@ticoann , @vkuznet
Shengchan, can you merge this PR? Also, can you copy recent  fwjr to vocms013 that has bunch of free storage?
Valentin,
couchDB seems not to include in the software sets on vocms013. Instructions on installing couchDB cmsweb based node?
Thanks,
Yuyi

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191363081

@vkuznet , @ticoann
If I remembered correctly, the fwjr copied from production nodes is in couchDB and I need to read them from couchDb in order to insert into MongoDB using WMArchive format.  Seangchan, please correct me if I am wrong.

@yuyiguo, yes, you are right we need couch server to retract the data from fwjr. Didn't we get the machine for this at fnal before?

Yuyi,
I still don't understand why do you need couchdb RPMs? We don't need
couch DB on vocms013 since you can read docs from couch via HTTP(s) anyhow.
So, I would assume your python (pseudo)-code will do that:
import urllib, urllib2
url = "YOUR COUCH URL"
params = {} # your dict of what you search for
req = urllib2.Request(url, urllib.urlencode(params))
response = urllib2.urlopen(req)
result = json.load(response.read())

import pymongo
client = pymongo.MongoClient()
# some code to insert data into mongo

This code will read data from couch, but you don't really need a couchdb.
On  0, Yuyi Guo<EMAIL_ADDRESS>wrote:

@vkuznet , @ticoann
If I remembered correctly, the fwjr copied from production nodes is in couchDB and I need to read them from couchDb in order to insert into MongoDB using WMArchive format.  Seangchan, please correct me if I am wrong.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191388484

Hi Valentin, since this is production data we don't want to directly read from couch. Doesn't want to  overload the production agent. So I was suggesting copy the file directly to other machine and run the couch server there.  @yuyiguo, Yuyi whether are you going to transfer the data (to Mongo db) vocms013?
We can use one of the fnal test machine how ever if you need to transfer to cern machine firewall has to be open.

@ticoann , @vkuznet
Seangchan,
The MongoDB is on vocms013. It would be nice to copy the data to vomcms013  and setup a CouchDb on vomcs013 directly. FNAL now has very restrict rules on open a port to outside world. Since our cmsdag machines are not supposed to open ports to outside FNAL. We may need a lot of paper work to do so. I thought it is easy to just install a couchDB on vocms013 directly.

Hmm, why not to use reqmgr2-dev then? Of course we can build WMArchive with couchdb
But a lot of stuff we don't really need, e.g. erlang, spidermonkey, couch.
Then we need to configure node to setup couch service. Can we just use couch
on existing dev VMs, e.g. reqmgr2-dev?
On  0, ticoann<EMAIL_ADDRESS>wrote:

Hi Valentin, since this is production data we don't want to directly read from couch. Doesn't want to  overload the production agent. So I was suggesting copy the file directly to other machine and run the couch server there.  @yuyiguo, Yuyi whether are you going to transfer the data (to Mongo db) vocms013?
We can use one of the fnal test machine how ever if you need to transfer to cern machine firewall has to be open.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191415338

Yuyi,
the couch is a back-end and it comes as dependency for some service.
In order to install couch, I need to make WMArchive depend on it.
Then I need to build WMArchive RPMs with it, install them.
Start separately couch. As I wrote I rather prefer to use existing
VMs where couch comes naturally, e.g. reqmgr2-dev.
My understanding we only need couch once to read the data.
On  0, Yuyi Guo<EMAIL_ADDRESS>wrote:

@ticoann , @vkuznet
Seangchan,
The MongoDB is on vocms013. It would be nice to copy the data to vomcms013  and setup a CouchDb on vomcs013 directly. FNAL now has very restrict rules on open a port to outside world. Since our cmsdag machines are not supposed to open ports to outside FNAL. We may need a lot of paper work to do so. I thought it is easy to just install a couchDB on vocms013 directly.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191418593

reqmgr2-dev is fine with me assuming it has enough storage to hold the data.

reqmgr2-dev doesn't have enough space. Yuyi, haven't you set up MongoDB at FNAL machine before?

some more unittest still need to be updated (will create the new issue) but since Yuyi need the code merging it.

The MongoDB is on vocms013. It would be nice to copy the data to vocms013 and setup a CouchDb on vomcs013 directly. FNAL now has very restrict rules on open a port to outside world. Since our cmsdag machines are not supposed to open ports to outside FNAL. We may need a lot of paper work to do so. I thought it is easy to just install a couchDB on vocms013 directly.

We don't need to open the port, do we? We will just push the data so vocms013 has to open the port but not FNAL machine.

I don't like any idea of opening ports. Why do we need to open vocms013 ports?
We should not fail security measurements, otherwise we'll be panished and node
will be revoked.
I can install manually everything, e.g. erlang and couch, if you need it to run
on vocms013.
On  0, ticoann<EMAIL_ADDRESS>wrote:

The MongoDB is on vocms013. It would be nice to copy the data to vocms013 and setup a CouchDb on vomcs013 directly. FNAL now has very restrict rules on open a port to outside world. Since our cmsdag machines are not supposed to open ports to outside FNAL. We may need a lot of paper work to do so. I thought it is easy to just install a couchDB on vocms013 directly.

We don't need to open the port, do we? We will just push the data so vocms013 has to open the port but not FNAL machine.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191429104

I can install manually everything, e.g. erlang and couch, if you need it to run on vocms013.

I know you can, just not to give you too much extra work. :smile:

@ticoann, @vkuznet
To me the best way is to install couchDB on vocms013, then we don't have security or space issue. Valenint, do you have instructions on install a stand alone couchDB? If so, please point them  to me. If you think it is easier to do it yourself instead of explaining them to me, go head please.

Yuiy, Seangchan,
please read vocms013:/data/wma/couchdb/HOWTO and build/compile/install/run your
own couch without messing around with WMArchive environment.
I encourage you to learn these basic skills to install manually any unix
package since it becomes handy in many circumstances. The basic idea is to
grab tarbal, untar, configure, make and make install.
For steps in HOWTO I added couple of system packages, e.g. ncurses and icu,
which were required by couch build.
Once you have your own couch in your own area you can put data in it.
Once you're done you may easily clean-up your own area without touch
WMArchive environment since I want to keep it clean (it will take more of
my time to clean messy environment or reinstall the node than build local
couch on my own).
Best,
Valentin.
On  0, Yuyi Guo<EMAIL_ADDRESS>wrote:

@ticoann, @vkuznet
To me the best way is to install couchDB on vocms013, then we don't have security or space issue. Valenint, do you have instructions on install a stand alone couchDB? If so, please point them  to me. If you think it is easier to do it yourself instead of explaining them to me, go head please.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191440196

Ok, I will build it .
Yuyi
Sent from my iPhone
On Mar 2, 2016, at 4:13 PM, Valentin Kuznetsov <notifications<EMAIL_ADDRESS>wrote:
Yuiy, Seangchan,
please read vocms013:/data/wma/couchdb/HOWTO and build/compile/install/run your
own couch without messing around with WMArchive environment.
I encourage you to learn these basic skills to install manually any unix
package since it becomes handy in many circumstances. The basic idea is to
grab tarbal, untar, configure, make and make install.
For steps in HOWTO I added couple of system packages, e.g. ncurses and icu,
which were required by couch build.
Once you have your own couch in your own area you can put data in it.
Once you're done you may easily clean-up your own area without touch
WMArchive environment since I want to keep it clean (it will take more of
my time to clean messy environment or reinstall the node than build local
couch on my own).
Best,
Valentin.
On 0, Yuyi Guo <notifications<EMAIL_ADDRESS>wrote:

@ticoann, @vkuznet
To me the best way is to install couchDB on vocms013, then we don't have security or space issue. Valenint, do you have instructions on install a stand alone couchDB? If so, please point them to me. If you think it is easier to do it yourself instead of explaining them to me, go head please.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-191440196

Reply to this email directly or view it on GitHubhttps://github.com/dmwm/WMCore/pull/6614#issuecomment-191456516.

@vkuznet , @ticoann
Thanks for the building instructions, Valentin.
Now couchdb is running on vocms013 under my area.

@ticoann
SeangChan, Below is the data from the production fwjr.
Besides PeakValueRss and PeakValueVsize needed to change the data type, EventThroughput is mssing in the schema as well as storage and multicore.
u'performance': {u'cpu': {u'AvgEventTime': u'10.3341',
u'EventThroughput': u'0.0952297',
u'MaxEventTime': u'48.3678',
u'MinEventTime': u'4.46389',
u'TotalJobCPU': u'1963.63',
u'TotalJobTime': u'2101.53',
u'TotalLoopCPU': u'1962.28'},
u'memory': {u'PeakValueRss': u'1528.52',
u'PeakValueVsize': u'1931.56'},
u'multicore': {},
u'storage': {u'readAveragekB': 77.8474891246,
u'readCachePercentageOps': 0.0,
u'readMBSec': 0.0438598972596,
u'readMaxMSec': 4832.84,
u'readNumOps': 97620.0,
u'readPercentageOps': 1.00032780168,
u'readTotalMB': 7423.792,
u'readTotalSecs': 0,
u'writeTotalMB': 357.624,
u'writeTotalSecs': 575158.0}},

Seangchan,
we also discussed with Yuyi that it is better to split fileArray into two
arrays. One will keep LFNs and another PFNs. The reason for that is simple
we mostly will look-up data by LFNs, therefore we can reduce its file array
by factor of 2 which will significantly impact on index size in DB.
Valentin.
On  0, Yuyi Guo<EMAIL_ADDRESS>wrote:

@ticoann
SeangChan, Below is the data from the production fwjr.
Besides PeakValueRss and PeakValueVsize needed to change the data type, EventThroughput is mssing in the schema as well as storage and multicore.
u'performance': {u'cpu': {u'AvgEventTime': u'10.3341',
u'EventThroughput': u'0.0952297',
u'MaxEventTime': u'48.3678',
u'MinEventTime': u'4.46389',
u'TotalJobCPU': u'1963.63',
u'TotalJobTime': u'2101.53',
u'TotalLoopCPU': u'1962.28'},
u'memory': {u'PeakValueRss': u'1528.52',
u'PeakValueVsize': u'1931.56'},
u'multicore': {},
u'storage': {u'readAveragekB': 77.8474891246,
u'readCachePercentageOps': 0.0,
u'readMBSec': 0.0438598972596,
u'readMaxMSec': 4832.84,
u'readNumOps': 97620.0,
u'readPercentageOps': 1.00032780168,
u'readTotalMB': 7423.792,
u'readTotalSecs': 0,
u'writeTotalMB': 357.624,
u'writeTotalSecs': 575158.0}},

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6614#issuecomment-192476925

Thanks Yuyi and Valentin, I created the new issue. #6670


request: add option to crop with modulo lengths to support effective stacking

For proper stacking of images generated by JSol'ex, one needs to have each frame to have the same dimension. With current version, cropping is only limited to (n)x of diameter, with the final sizes usually are different because the sun diameter may change from one to another. I know one can restrict the final cropped size to be closest to a multiplied value of a constant in ImageMath scripts, but it would be great if this option is also available in the default mode as well.

That makes two of us (at least). I requested this in #390. But I think you stated the request better.

Let's think about how to do this properly. With the current options:

no crop : all images will have different sizes
source width : all images will have different sizes
radius factor : most images should have the same size, but because the detected radius may be different, it can happen that the sizes are different
fixed width (non existent option) : all images would have the same width, but because the radius may be different, the alignment will not work, or the stacking will not work because the scale of the solar disk will be different

This leaves us with an option to first crop with whatever technique (preferably radius crop) then do modulo lengths. The problem I'm having with this is that ideally, you shouldn't have to determine which modulo to apply. This is what happens if you use JSol'Ex built-in stacking, it determines the sizes automatically and does the right job, but it can only do so because it has the list of images at hand and that the solar disks are known. In this case, we only have a single disk (single file processing). Even in batch mode, you don't know the sizes of the other disks before you have completed processing, so it would require re-cropping after processing the whole batch, which introduces an army of secondary problems, therefore a no-go for me. As such, the only reasonable option I see is specifying the "rounding" modulo manually (e.g 128, 256) and hope that it will work. Not great user XP though.

Agree with your conclusion that a manually entered rounding modulo is the
best compromise. Works for me.
On Thu, Aug 15, 2024, 3:08 AM Cédric Champeau @.***>
wrote:

Let's think about how to do this properly. With the current options:

no crop : all images will have different sizes
source width : all images will have different sizes
radius factor : most images should have the same size, but because
the detected radius may be different, it can happen that the sizes are
different
fixed width (non existent option) : all images would have the same
width, but because the radius may be different, the alignment will not
work, or the stacking will not work because the scale of the solar disk
will be different

This leaves us with an option to first crop with whatever technique
(preferably radius crop) then do modulo lengths. The problem I'm having
with this is that ideally, you shouldn't have to determine which modulo to
apply. This is what happens if you use JSol'Ex built-in stacking, it
determines the sizes automatically and does the right job, but it can only
do so because it has the list of images at hand and that the solar
disks are known. In this case, we only have a single disk (single file
processing). Even in batch mode, you don't know the sizes of the other
disks before you have completed processing, so it would require re-cropping
after processing the whole batch, which introduces an army of secondary
problems, therefore a no-go for me. As such, the only reasonable option I
see is specifying the "rounding" modulo manually (e.g 128, 256) and hope
that it will work. Not great user XP though.
—
Reply to this email directly, view it on GitHub
https://github.com/melix/astro4j/issues/408#issuecomment-2291014038, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BGFJTAVSW2UQX7TGC2WCGODZRR42TAVCNFSM6AAAAABMRBHLLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJRGAYTIMBTHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Cèdric:  I think the reason the detected radius is changing, is that it is close to a boundary between two autocrop choices. Either atmospheric distortion or a slight change in prominences, sends the width to one or the other adjacent values. A fixed width here, renders the crop algorithm insensitive to the boundary between two crop values: it fixes the choice - important for stacking.
Autocrop chooses width in multiples of 16 (in area of 256). If this were reserved for a fixed width option, it would be modulo 16 - unclear to me weather your use of modulo refers to width or area.
Addressing issue #408 would also address #390.

Could be the same as https://github.com/melix/astro4j/issues/416


feat(bindings/html): Accept Buffer|string instead of Buffer

Description:
Related issue (if exists):

@slorber Once it's published, @swc/html will accept string | Buffer for minify* APIs. I'll fix the publish pipeline asap and publish a new version soon.

Thanks :)


CSS Framework

#Framework
##User Story
As a developer, I want to ensure that the chosen CSS framework works.
##Acceptance Criteria
It is done when Materialize works well with HTML and Javascript.

https://materializecss.com/select.html


ARTEMIS-2753 Fixing OpenWire Temporary queue names over wildcard

Im just looking at this, on windows it seems it doesnt pick up. Another note re wildcards looks like something might be up with Advisories, we dont have many test in that space though.... so i havent got much to steal and test with,....unless you know of somewhere and can point me to?

@michaelpearce-gain  I don't think this is a real issue in most environments... however since I have tests failing now... I wanted to be diligent and make sure it's fixed.


Command Pallete Shortcut is not being intercepted on Firefox

code-server version: 1.32.0-310
OS Version: CoreOS build 2025

Description
On Firefox Quantum, Coder does not intercept the command pallete shortcut which is the private browsing shortcut as well on Firefox. Even Alt + KEY refuses to work.
Steps to Reproduce

Open Firefox
Hit the command pallete shortcut

@sr229 Does F5 work? (should still be fixed for the other one)

I don't believe we can override this.

I think folks at Theia found a way around this, worth a look.

To be sure I made a small test on an empty webpage binding keydown and preventing default and stopping propagation on the event but had no luck.
I searched around Theia's GitHub and didn't find anything except this open issue: https://github.com/theia-ide/theia/issues/1696. As mentioned there, F1 is a viable workaround, but it doesn't appear possible to override ctrl+shift+p.


Fix polls bug

Problem

Polls with possible multiple answers not working

Solution

Fixed in upstream dev branch - Use this branch as source instead of release this time

PR Status

[x] Code finished and ready to be reviewed/tested
[ ] The fix/enhancement were manually tested (if applicable)

Automatic tests
Automatic tests can be triggered on https://ci-apps-dev.yunohost.org/ after creating the PR, by commenting "!testme", "!gogogadgetoci" or "By the power of systemd, I invoke The Great App CI to test this Pull Request!". (N.B. : for this to work you need to be a member of the Yunohost-Apps organization)

!testme


Revert "chore(deps): update jamesives/github-pages-deploy-action digest to a96ffa2"

Reverts detekt/detekt#7680.
Reason: https://github.com/detekt/detekt/actions/runs/11077673195/job/30783459115#step:9:22
Don't merge new versions of this action until these are addressed:

https://github.com/JamesIves/github-pages-deploy-action/issues/1698
https://github.com/JamesIves/github-pages-deploy-action/issues/1697

v4.6.4 is the last known good version.

Superseded by #7687


Whiteout files not supported

When a whiteout file is created it should result in a file being removed from the resulting fs. For example:
$ cd $(mktemp -d)
$ echo "repo contents" > foo.repo
$ touch .wh.foo.repo
$ cat <<DF > Dockerfile
FROM fedora:28

ADD foo.repo /opt/foo.repo
RUN cat /opt/foo.repo
ADD .wh.foo.repo /opt/
RUN cat /opt/foo.repo
DF
$ docker build .
Sending build context to Docker daemon 3.584 kB
Step 1/5 : FROM fedora:28
 ---> cc510acfcd70
Step 2/5 : ADD foo.repo /opt/foo.repo
 ---> Using cache
 ---> fe5009f4a07d
Step 3/5 : RUN cat /opt/foo.repo
 ---> Using cache
 ---> 282b0d8f9bd2
Step 4/5 : ADD .wh.foo.repo /opt/
 ---> Using cache
 ---> 6d064295b23e
Step 5/5 : RUN cat /opt/foo.repo
 ---> Running in 9a82b576101a
cat: /opt/foo.repo: No such file or directory
The command '/bin/sh -c cat /opt/foo.repo' returned a non-zero code: 1

The command fails because the whiteout file takes effect. With imagebuilder in contrast:
$ imagebuilder -t foo .
--> FROM fedora:28 as 0
--> ADD foo.repo /opt/foo.repo
--> RUN cat /opt/foo.repo
repo contents
--> ADD .wh.foo.repo /opt/
--> RUN cat /opt/foo.repo
repo contents
--> Committing changes to foo ...
--> Done
[tmp.grxZVQQgyo] $ docker run foo ls -la /opt/
total 4
drwxr-xr-x. 1 root root 22 May 24 18:03 .
drwxr-xr-x. 1 root root  6 May 24 18:04 ..
-rw-rw-r--. 1 root root 14 May 24 17:21 foo.repo

The file remains. This is not really a difference in docker build, it's simply that whiteout files act on previous layers and with imagebuilder the whiteout file is placed in the same layer as the file it was intended to affect.
It's arguable whether we really want imagebuilder to emulate this somehow, but it would really simplify this use case.

Different storage drivers have different ways to handle removal of contents that were present in a lower layer.  What's the intended result of creating a .wh.foo.repo file instead of just deleting foo.repo?

Fair point, it seemed like a hack anyway.
Deleting a file (let's assume with RUN rm file) requires that the current USER have permission to delete the file, so it's not a simple inverse of ADD. In our case we're wanting to automate the addition of a yum repo file at the beginning of the Dockerfile and then removing it at the end (to supply a build repo written on the fly). This means we have to determine what the USER is supposed to be at the end and switch it like so:
USER root
RUN rm /etc/yum.repos.d/foo.repo
USER previous_user

And that assumes rm is even present -- if your base image is scratch that will break (so would using yum, but  this is automation, it doesn't know if the repo will be needed or not).
As an alternative, we can ADD an empty file to overwrite it at the end, but then you have a useless empty file to confuse people. Ideally, we could mount the file as a volume during the build so it's never present in the container; but the repo file is generated in a container fs, so mounting it into the build container is complicated.
It would be great if there were a REMOVE instruction or ADD --remove option for removing files, or some other way for imagebuilder to facilitate this.
But if nothing else, this issue just serves as notice that the whiteout file hack works differently here.

I think docker actually standardizes on the whiteout format inside the tar.  The question is whether we can safely do that via the invocation of the command.  You'd get into some interesting cases where whiteout doesn't have the same behavior.


Make de NFS provisioner configuration optional

As an user deploying my own cluster I would like to avoid getting the NFS provisioner configured out of the box without explicitly setting it via vat.tf. In a production deployment I will probably use an already existing or a new NFS server and will  include some customization for my deployment to meet some requirements.
I had to execute the following before installing a customized one:
oc delete all -l app=nfs-client-provisioner -n nfs-provisioner
oc delete project nfs-provisioner

https://github.com/ocp-power-automation/ocp4-upi-powervs/blob/master/variables.tf#L393-L397 If you do not set any value or something other than "nfs" then the automation will not create an additional volume or configure storage for you.

It should not even install the provider if not explicitly required by the user. For instance, I didn't set any NFS parameter and when I was about to install a custom provider I had to first remove what was installed. All NFS related items should be installed only if the user set this as true.

Can you please provide more information as what NFS resources were created when you did not set storage_type to "nfs" (this is by default)?
This is the o/p from a HA cluster on PowerVS which does not use NFS storage configurations:
# oc get project nfs-provisioner
Error from server (NotFound): namespaces "nfs-provisioner" not found

Can you please provide more information as what NFS resources were created when you did not set storage_type to "nfs" (this is by default)?
This is the o/p from a HA cluster on PowerVS which does not use NFS storage configurations:
# oc get project nfs-provisioner
Error from server (NotFound): namespaces "nfs-provisioner" not found

@rpsene @yussufsh is this still relevant or can be closed ?

@rpsene @yussufsh is this still relevant or can be closed ?

@bpradipt this is one the "saving time" options that we need to address. The NFS server can be set in a new VM, outside the Bastion and be configured in parallel. Making it option is still a feature request to give the user the option to enable it or not, or even use something already available.

@bpradipt this is one the "saving time" options that we need to address. The NFS server can be set in a new VM, outside the Bastion and be configured in parallel. Making it option is still a feature request to give the user the option to enable it or not, or even use something already available.


julia-client install failed ENOENT: no such file or directory, open '...\apm-install-dir-120124-35008-40fjld.jxg3x\package.json'

Details

Atom version: 1.440
Julia version: 1.3.1
OS: Windows 10 64bit
Package versions:

Atom.jl: 0.12.5
julia-client: 0.12.2
ink: 0.12.2

Steps to reproduce
Uninstalled all Julia related packages from Atom.
Installed Julia version 1.3.1
Installed Atom and Juno packages
Then...
Microsoft Windows [Version 10.0.18362.657]
(c) 2019 Microsoft Corporation. All rights reserved.
C:\Users\Spock>apm install ink
Installing ink to C:\Users\Spock.atom\packages done
C:\Users\Spock>apm install language-julia
Installing language-julia to C:\Users\Spock.atom\packages done
C:\Users\Spock>apm install julia-client
Installing julia-client to C:\Users\Spock.atom\packages failed

<EMAIL_ADDRESS>install C:\Users\Spock\AppData\Local\Temp\apm-install-dir-120124-35008-40fjld.jxg3x\node_modules\julia-client\node_modules\node-pty-prebuilt-multiarch
prebuild-install || node scripts/install.js

C:\Users\Spock\AppData\Local\Temp\apm-install-dir-120124-35008-40fjld.jxg3x\node_modules\julia-client\node_modules\node-pty-prebuilt-multiarch>if not defined npm_config_node_gyp (node "C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
prebuild-install WARN install unexpected end of file
gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at PythonFinder.failNoPython (C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\node-gyp\lib\configure.js:492:19)
gyp ERR! stack     at PythonFinder. (C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\node-gyp\lib\configure.js:517:16)
gyp ERR! stack     at C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\graceful-fs\polyfills.js:284:29
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:182:21)
gyp ERR! System Windows_NT 10.0.18362
gyp ERR! command "C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\bin\node.exe" "C:\Users\Spock\AppData\Local\atom\app-1.44.0\resources\app\apm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\Spock\AppData\Local\Temp\apm-install-dir-120124-35008-40fjld.jxg3x\node_modules\julia-client\node_modules\node-pty-prebuilt-multiarch
gyp ERR! node -v v10.2.1
gyp ERR! node-gyp -v v3.7.0
gyp ERR! not ok
npm WARN enoent ENOENT: no such file or directory, open 'C:\Users\Spock\AppData\Local\Temp\apm-install-dir-120124-35008-40fjld.jxg3x\package.json'
npm WARN apm-install-dir-120124-35008-40fjld.jxg3x No description
npm WARN apm-install-dir-120124-35008-40fjld.jxg3x No repository field.
npm WARN apm-install-dir-120124-35008-40fjld.jxg3x No README data
npm WARN apm-install-dir-120124-35008-40fjld.jxg3x No license field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>install: prebuild-install || node scripts/install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\Spock.atom.apm_logs\2020-02-24T06_59_07_437Z-debug.log
I've attached the complete log file.
log.txt

Hm, prebuild-install WARN install unexpected end of file sounds like you got a corrupted file (or your antivirus software tried to delete it). Does this happen every time you try installing julia-client?

This is happening every time I try to install julia-client. I didn't encounter this issue with previous versions of julia-client on the same machine. I'm not running any particular anti-virus software such as Norton, Symantec etc.
 Can't find Python executable "python", you can set the PYTHON env variable

Should I be defining the PYTHON env variable?

The current setup tries to download prebuilt binaries from github, which -- for some reason -- fails for you. As a fallback we try building the binary on your machine, which is kinda fiddly to set up.
So yes, you can try to point PYTHON to a Python 2 instance after following the installation instructions for node-pty.

Thanks, I will give it a go tomorrow once I am back at that particular machine.

Unfortunately, all my attempts at getting this to work so far have failed. It seems that a TEMP directory containing the julia-client scripts etc. gets deleted before its install scripts are executed.
2391 verbose lifecycle<EMAIL_ADDRESS>CWD: C:\Users\Spock\AppData\Local\Temp\apm-install-dir-120128-2504-l39f64.esq4b\node_modules\julia-client\node_modules\node-pty-prebuilt-multiarch 2392 silly lifecycle<EMAIL_ADDRESS>Args: [ '/d /s /c', 'prebuild-install || node scripts/install.js' ] 2393 silly lifecycle<EMAIL_ADDRESS>Returned: code: 1  signal: null 2394 info lifecycle<EMAIL_ADDRESS>Failed to exec install script
I tried to set everything up to build but it is failing with error messages that I can't make sense of.
I ran apm install julia-client --verbose and got pages of error messages for winpty.vcproj that complain about unexpected tokens etc. as if an incorrect compiler is being used. It says things such as This version of node/NAN/v8 requires a C++11 compiler.
I've attached the verbose output, as well as the concomitant log file.
verbose_error.txt
log.txt
Thanks for any help you might be able to offer.

I think that this may be an issue with Atom 1.44.0. I deleted all my atom folders, including

C:\Users<user name>.atom
C:\Users<user name>\AppData\Local\atom
C:\Users<user name>\AppData\Roaming\Atom

I then downloaded the atom-windows.zip https://github.com/atom/atom/releases/tag/v1.42.0 (should have actually downloaded the x64 version instead) and extracted it.
Upon launching that version of Atom, I installed uber-juno which installed the relevant packages without any problems. Because Atom is now a "local" install it seems like I no longer have apm.
I will try a similar procedure with newer versions of a "local" Atom installation next week. Maybe something got messed up with apm itself.

Yes, it's likely that the global apm install got messed up somehow. Some people have reported that apx got around the installation issues, so you could give that a go as well.


Compile error when using append in a Realm List (Swift 3)

Goals
I want to be able to add multiple objects to a Realm List.
Expected Results
Code compiles without errors.
Actual Results
Unable to compile both variants of the code.
project.messages.append(contentsOf: foundMessages) gives me following error: Extraneous argument label 'contentOf:' in call
But when I remove the extraneous argument I get following error:
project.messages.append(foundMessages), Cannot convert value of type '[Message]' to expected argument type 'Message'
Code Sample
import Foundation
import RealmSwift

class Demo {
  func test() {
    let house1 = House()
    let house2 = House()
    let village = Village()
    village.houses.append(contentsOf: [house1, house2])
    //village.houses.append([house1, house2])
    // both variants cause a compiler error
  }
}

class House: Object {
}

class Village: Object {
  let houses = List<House>()
}

Version of Realm and Tooling
ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G31
/Applications/Xcode.app/Contents/Developer
Xcode 8.2
Build version 8C38
/usr/local/bin/pod
1.1.1
Realm (2.1.1)
RealmSwift (2.1.1)
RealmSwift (~> 2.1)
/bin/bash
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)
/usr/local/bin/carthage
0.17.2
(not in use here)
/usr/bin/git
git version 2.10.1 (Apple Git-78)

@D-32 Use append(objectsIn:) instead append(contentsOf:).
...
village.houses.append(objectsIn: [house1, house2])
...

That's an easy fix, works! :)
I didn't get any warning about it being deprecated though...


Buildalizer chooses MSBuild without .NET components

I have the following Visual Studio installations:

Visual Studio 2019 (with .NET components)
Visual Studio Build Tools 2019 (with C++ components only)

Buildalizer chooses VS Build Tools 2019 installation to run msbuild. Since it has no .NET components, the build fails. Installing .NET components to that installation fixes the problem. But maybe it's possible to choose the appropriate installation by a Buildalizer itself?

What an interesting scenario. It never occurred to me this might be a problem, but it makes total sense. I guess the challenge is knowing which one to pick - I.e. if it were reversed and the VS Build Tools had the .NET components but the VS install only had C++, how would we know?

What an interesting scenario. It never occurred to me this might be a problem, but it makes total sense. I guess the challenge is knowing which one to pick - I.e. if it were reversed and the VS Build Tools had the .NET components but the VS install only had C++, how would we know?

vswhere allows to search for VS installation with required components: https://github.com/microsoft/vswhere
I guess it can be used to find appropriate installation. It's a part of VS Installer, so it's possible to call it from there. Also, it looks like there is an option to distribute vswhere as a part of Buildalizer.

vswhere allows to search for VS installation with required components: https://github.com/microsoft/vswhere
I guess it can be used to find appropriate installation. It's a part of VS Installer, so it's possible to call it from there. Also, it looks like there is an option to distribute vswhere as a part of Buildalizer.


"Event Edit" / "Event Create" page for organizer

Todo:

[ ] Add label to use-group switch
[x] Only show relevant fields depending on action
[x] Add actual inputs for date and time
[ ] Add logic to event-edit page
[ ] Add client-side validation
[ ] Improve styling
[ ] Improve error handling

What stuff needs to be done here?


higher order moments in hillas parameter version 2

hillas_parameters_1 and hillas_parameters_2 are now compatible in units

the hillas parameters got moved. You need to
git fetch upstream
git rebase upstream/master

To resolve the conflicts.
I think this is needed quite urgend, because the missing units is one of the last reasons the docs fail to build (issue warnings).

Now, something ist seriously broken. I do not know how. But you added the submodules themselves to the repository.

The problem is the second to last commit: 87396e8
I think it is best do remove it completely from the history.
What you could do (but which is kind of dangerous and you should back up your local copy of the repo before) is:
git reset --hard 8e547f8  # go back to the last commit before the strange stuff happend
git push -f

I removed the problematic commits! Now I do the following -
git pull upstream master # upstream is alias to the cta-observatory repo  git submodule update  git push -u origin master  #master is alias to my github repo
But ctapipe-extra still seems to be the old version.

Thats because you commited the change in submodule and there was no conflict with the master, so git kept your change.
You have to update back to the version it was before.
 Can you describe what you are doing and in which directory?

The directory structure is cta_dir/ctapipe. Inside this I do git init. There are two remotes - origin (my github repo) , master (cta-observatory repo). cta_dir/ctapipe contains many directories including ctapipe, ctapipe-extra, astropy-helpers. In cta_dir/ctapipe I do - `cd ctapipe-extra
git checkout f7fe2209646dc8f99a28ec9ab4df8257687e1896
cd ..
git add ctapipe-extra
git commit -m 'Revert ctapipe-extra downgrade'
git submodule update`
After this from my local copy cta_dir/ctapipe/ctapipe everything gets deleted with just an empty 'image' folder.
And inside cta_dir/ctapipe two new directories ctapipe-extra and datasets gets created!

Mmmh. This sounds strange.
Ok, let's fix this strange commit first.
For this, go to the cta_dir/ctapipe directory. The one that contains the setup.py and the  ctapipe-extra and astropy_helpers directories.
Here we remove the last two commits by going back to the commit with the hash 8e547f864c861e7c259abcd11aad10fa34ad83fc:
git reset --hard 8e547f864c861e7c259abcd11aad10fa34ad83fc

After that, you need to do
git push -f

The -f is required because you delete commits from the online repository.

The problematic commit removed. Now I just start from the scratch. Now if I do a clone of my git hub repo, the directories ctapipe-extra and astropy_helpers are empty in my local copy!
Although I can see them in the online github repo.

Yes. Now you can do
git submodule init
git submodule update

The repositories are filled then.

Yes! Now git push ?

There should be nothing you can push right now. We only fully cloned the repo with all submodules.
Now we need to update the ctapipe-extra version to the one in master.
ah, and i think i found the mistake.
Could you do
cd ctapipe-extra
git checkout f7fe2209646dc8f99a28ec9ab4df8257687e1896
cd ..

And tell me the output of git status?

On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)
modified:   ctapipe-extra (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

Now ctapipe-extra looks updated in the local copy.

If you now do git add ctapipe-extra what does git diff --staged say?

 Phew! Thank you very much.

Your changes to the hillas_parameter function broke the camera_display example and probably also a few others. This is why the documentations fails to build. Can you adapt the examples?

Now the hillas_parameters code returns two things - MomentParameters and HighOrderMomentParameters (two namedtuples). But the camera_display example now works with only one return value and hence cannot recognize two returns.
How to resolve this? Should a single namedtuple be used instead of two?
The interactive_test_hillas.py code in ctapipe/image/tests handles the two return values correctly for now.

I think both versions of the hillas_parameters should return a single Container, probably a new HillasContainer, if I understood the current return value concept correctly.
@kosack

I just added a preliminary version of HillasContainer for demo of storing the hillas_parameters along with an interactive test script. A single Container is used to store the MomentParameters and HighOrderMomentParameters.

Okay! Now I have committed changes to use a single HillasContainer holding all the shower moments as items. The camera_display example should work now.

Container looks good. However, you downgraded the astropy helpers. Just never add changes in astropy_helpers or ctapipe_extra if it's not your intention do upgrade or downgrade them. Ypu should revert 	80eaa20:
git revert 	80eaa20
git push

There are some markup errors in the doc strings. I would also remove the change log in the file. This is what we have git for.

I fixed some of the doc string errors that I could identify manually, but make doc did not show any doc string warnings/errors for hillas.py.

I think both versions of the hillas_parameters should return a single Container, probably a new  HillasContainer, if I understood the current return value concept correctly.

The idea of the Containers is that they are just table definitions for output files, so when we do a hillas analysis, they should include anything that we want to be written to the output table (so all Hillas parameters in this case).  I'd say the higher-order moments are always useful, so there is no reason to separate them into two.

I think this is all super-ceded by other commits. All hillas algorithms (4 of them!) seem to be in agreement and have common high-order outputs.


Specify SaveOptions in server capabilities

Similar to https://github.com/Pure-D/serve-d/pull/320, this PR adds SaveOptions to ther server capabilities, to make some editors send a textDocument/didSave event.
I confirmed the fix with helix version 23.10.
includeText is set to false as I think the text is not used, but I am not familiar with this projects code.
Relevant docs: https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_didSave
Related issue on Helix side: https://github.com/helix-editor/helix/pull/3957

@dmorawetz, this might not be the correct solution as it is breaking a lot of end-to-end tests with our supported VS Code Ansible extension. Please look into it.

Sorry for the late reply, but I am quite busy right now. I will take a look at it. Hopefully soon.


UR5 not moving when modern_driver used in a docker container on a bridge network

I use this package with my universal robot UR5 in a docker container.
Everything is working fine when the container is registered on the host network but I cannot get the robot moving when the container lives on the default bridge network created by docker.
This problem has been mentionned #111 and #118 but not properly solved yet.
If anybody knows how to configurate the bridge network to enable correct communication between the container and the UR5, I'll be very thankfull.
Thx for your help

Problem solved, see end of #111 for the solution.


Compilation error

monaco-editor version: 0.8.2
OS: Windows
Before fallow  the steps here When I try do:
# start the compiler in the background
/src/vscode> node_modules/.bin/gulp watch-client
This error appears.
[10:07:01] Error: C:/Users/root/Desktop/Monaco/vscode/src/vs/platform/telemetry/node/workbenchCommonProperties.ts(6,25): Cannot find module 'winreg'.
[10:07:01] Finished compilation with 1 errors after 69894 ms
[10:07:06] [monaco.d.ts] Starting monaco.d.ts generation
[10:07:08] [monaco.d.ts] Finished monaco.d.ts generation
[10:07:09] Starting compilation...
[10:07:26] Error: C:/Users/root/Desktop/Monaco/vscode/src/vs/platform/telemetry/node/workbenchCommonProperties.ts(6,25): Cannot find module 'winreg'.
[10:07:26] Finished compilation with 1 errors after 17188 ms
I success finished this steps:
/src> git clone https://github.com/Microsoft/vscode
# install minimal npm deps for vscode
/src/vscode> npm run monaco-editor-setup

I'm sorry I tried really hard to have a subset of the vscode build tools, but this is not maintainable. I've updated the guide to include that the entire vscode pre-requisites are needed.


Is it possible to chain middleware ?

As described in Silex documentation, there's a finish middleware that can execute tasks after the Response has been sent to the client.
How could we do it for a particular route? Say for example: /crud/myTable/update/xxx

Hi, sry for the late answer. :)
You can add middlewares to routes after mounting like this:
$app->flush();
$route = $app['routes']->get('crudList');
$route->before(function(Symfony\Component\HttpFoundation\Request $request) use ($app) {
    // ...
});

Existing routes of CRUDlex are:

static (I just see, this should be prefixed, I will open another issue for this)
crudCreate
crudList
crudShow
crudEdit
crudDelete
crudRenderFile
crudDeleteFile
crudSetLocale

I will also add this to the documentation as it might be usefull. :)
Feel free to reopen if you have any further questions.

(Static route prefix done: https://github.com/philiplb/CRUDlex/commit/889dbad3ab2dea896492a67cbcd59f07049a85b9)

I added some documentation about it: http://philiplb.github.io/CRUDlex/docs/html/0.12.0/manual/routes.html

Thanks a lot


Update Docs: Intro “add-to-your-site”

Automatically generated by Netlify CMS

I added this for two reasons, one to test Netlify CMS and secondly to add docusaurus!


Add all_fields and auto support in the pydantic integration

Description
This adds support for the auto type annotation described in #1192 to the Pydantic integration, which allows a user to define the list of fields without having to re-specify the type themselves. This gives better editor and type checker support (although I didn't update the Mypy integration).
I haven't marked the fields list as deprecated, I wasn't sure if we wanted to do that in this release or not, but that's an easy change.
I also did a bit of cleanup on the error_type section, since it was missing a lot of the logic in the type function that allows for more complex input types.
Types of Changes

[ ] Core
[ ] Bugfix
[x] New feature
[x] Enhancement/optimization
[x] Documentation

Issues Fixed or Closed by This PR

Closes #1192

Checklist

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Thanks for adding the RELEASE.md file!

Here's a preview of the changelog:

Adds support for the auto type annotation described in #1192 to the Pydantic
integration, which allows a user to define the list of fields without having to
re-specify the type themselves. This gives better editor and type checker
support. If you want to expose every field you can instead pass
all_fields=True to the decorators and leave the body empty.
import pydantic
import strawberry
from strawberry.experimental.pydantic import auto

class User(pydantic.BaseModel):
    age: int
    password: str

@strawberry.experimental.pydantic.type(User)
class UserType:
    age: auto
    password: auto

Here's the preview release card for twitter:

Here's the tweet text:
🆕 Release (next) is out! Thanks to Matt Allen for the PR 👏

Get it here 👉 https://github.com/strawberry-graphql/strawberry/releases/tag/(next)


Add timestamp

This probably should be marked as Draft/WIP until the dependent change, SUSE-Enceladus/csp-billing-adapter#131 has merged.


Provisioning tools

Here is VS/MSBuild extensions that help working with provisioning templates. Basically it can contains a custom project template "SharePoint PnP Site Template" that has custom property page (where you can set template properties) and custom build target that build .pnp package. There is now smooth installer currently.
It may also be merged with Provisioning.VSTools solution (some code may be reused across both solutions). I just don't have an idea how to do it better, so I'll submit in a separate solution now.
Br, Ivan

Hi @ivanvagunin, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

thx Ivan, we'll have a look on this soon and get things moving.

great, thanks!

Hi Ivan, Would you be up for demonstrating this tomorrow in the Core, PowerShell and Engine SIG? - Would get recorded and would be beneficial for the field to see this in action from your machine. .NET core / PowerShell SIG is at 4 PM Central European Time tomorrow.

Hi Vesa, I am totally up for it.
On Jan 24, 2017 8:34 PM, "Vesa Juvonen"<EMAIL_ADDRESS>wrote:

Hi Ivan, Would you be up for demonstrating this tomorrow in the Core,
PowerShell and Engine SIG? - Would get recorded and would be beneficial for
the field to see this in action from your machine. .NET core / PowerShell
SIG is at 4 PM Central European Time tomorrow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SharePoint/PnP-Tools/pull/107#issuecomment-274893834,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIWp7ou8ye8zJpOhJUTVoNzIpSH7JuBpks5rVkQtgaJpZM4Lq0sd
.

thx Ivan, it's then go based on this. I'll get you added on the agenda as community demo. Thx for doing this advance. Cool tool.

thx Ivan for these. Getting things merged.


Cannot login on msal-react, error on console.log()

Core Library
MSAL.js v2 (@azure/msal-browser)
Core Library Version
^2.14.2
Wrapper Library
MSAL React (@azure/msal-react)
Wrapper Library Version
MSAL.js for React B2C Sample
Description
I am getting an error in the console.log() FetchClient.ts:24   as the place for the error. Line 24 is the await in the function below
/**
* Fetch Client for REST endpoints - Get request
* @param url
* @param headers
* @param body
*/
async sendGetRequestAsync(url: string, options?: NetworkRequestOptions): Promise<NetworkResponse> {
let response;
try {
response = await fetch(url, {
method: HTTP_REQUEST_TYPE.GET,
headers: this.getFetchHeaders(options)
});
} catch (e) {
if (window.navigator.onLine) {
throw BrowserAuthError.createGetRequestFailedError(e, url);
} else {
throw BrowserAuthError.createNoNetworkConnectivityError();
}
}
Error Message
The resulting error from the console.log();
GET https://promise%20ltd.b2clogin.com/promise%20ltd.onmicrosoft.com/b2c_1_signin_signout/v2.0/.well-known/openid-configuration net::ERR_NAME_NOT_RESOLVED
Msal Logs
not sure
MSAL Configuration
// Config object to be passed to Msal on creation
export const msalConfig = {
    auth: {
        clientId: "3a68f70e-c7c9-4115-9b55-7bf1da0d366f",
        authority: "https://Promise Ltd.b2clogin.com/Promise Ltd.onmicrosoft.com/B2C_1_signin_signout",
        knownAuthorities: ["Promise Ltd.b2clogin.com"],
        redirectUri: "http://localhost:4200",
        postLogoutRedirectUri: "http://localhost:4200"
    }
};

// Scopes you add here will be prompted for consent during login
export const loginRequest = {
    scopes: ["https://Promise Ltd.onmicrosoft.com/helloapi/demo.read"]
};

Relevant Code Snippets
/*
 * Copyright (c) Microsoft Corporation. All rights reserved.
 * Licensed under the MIT License.
 */

import { INetworkModule, NetworkRequestOptions, NetworkResponse } from "@azure/msal-common";
import { BrowserAuthError } from "../error/BrowserAuthError";
import { HTTP_REQUEST_TYPE } from "../utils/BrowserConstants";

/**
 * This class implements the Fetch API for GET and POST requests. See more here: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
 */
export class FetchClient implements INetworkModule {

    /**
     * Fetch Client for REST endpoints - Get request
     * @param url
     * @param headers
     * @param body
     */
    async sendGetRequestAsync<T>(url: string, options?: NetworkRequestOptions): Promise<NetworkResponse<T>> {
        let response;
        try {
            response = await fetch(url, {
                method: HTTP_REQUEST_TYPE.GET,
                headers: this.getFetchHeaders(options)
            });
        } catch (e) {
            if (window.navigator.onLine) {
                throw BrowserAuthError.createGetRequestFailedError(e, url);
            } else {
                throw BrowserAuthError.createNoNetworkConnectivityError();
            }
        }

        try {
            return {
                headers: this.getHeaderDict(response.headers),
                body: await response.json() as T,
                status: response.status
            };
        } catch (e) {
            throw BrowserAuthError.createFailedToParseNetworkResponseError(url);
        }
    }

    /**
     * Fetch Client for REST endpoints - Post request
     * @param url
     * @param headers
     * @param body
     */
    async sendPostRequestAsync<T>(url: string, options?: NetworkRequestOptions): Promise<NetworkResponse<T>> {
        const reqBody = (options && options.body) || "";

        let response;
        try {
            response = await fetch(url, {
                method: HTTP_REQUEST_TYPE.POST,
                headers: this.getFetchHeaders(options),
                body: reqBody
            });
        } catch (e) {
            if (window.navigator.onLine) {
                throw BrowserAuthError.createPostRequestFailedError(e, url);
            } else {
                throw BrowserAuthError.createNoNetworkConnectivityError();
            }
        }

        try {
            return {
                headers: this.getHeaderDict(response.headers),
                body: await response.json() as T,
                status: response.status
            };
        } catch (e) {
            throw BrowserAuthError.createFailedToParseNetworkResponseError(url);
        }
    }

    /**
     * Get Fetch API Headers object from string map
     * @param inputHeaders
     */
    private getFetchHeaders(options?: NetworkRequestOptions): Headers {
        const headers = new Headers();
        if (!(options && options.headers)) {
            return headers;
        }
        const optionsHeaders = options.headers;
        Object.keys(optionsHeaders).forEach((key) => {
            headers.append(key, optionsHeaders[key]);
        });
        return headers;
    }

FetchClient.ts
-----------------------

    private getHeaderDict(headers: Headers): Record<string, string> {
        const headerDict: Record<string, string> = {};
        headers.forEach((value: string, key: string) => {
            headerDict[key] = value;
        });
        return headerDict;
    }
}

Reproduction Steps
https://github.com/AzureAD/microsoft-authentication-library-for-js/tree/dev/samples/msal-react-samples/b2c-sample
Expected Behavior
this error is not supposed to show
Identity Provider
Azure B2C Basic Policy
Browsers Affected (Select all that apply)
Internet Explorer
Regression
none
Source
Internal (Microsoft)

Hi @getwebem ! Please review your MSAL and request configuration, the authority and knownAuthorities values seem off to me, same as the initial part of the scope, these should be normal URLs. There should be no spaces, I believe these values are incorrect.
// Config object to be passed to Msal on creation
export const msalConfig = {
    auth: {
        clientId: "3a68f70e-c7c9-4115-9b55-7bf1da0d366f",
        authority: "https://Promise Ltd.b2clogin.com/Promise Ltd.onmicrosoft.com/B2C_1_signin_signout", // There shouldn't be any spaces in this string
        knownAuthorities: ["Promise Ltd.b2clogin.com"], // There shouldn't be any spaces in this string
        redirectUri: "http://localhost:4200",
        postLogoutRedirectUri: "http://localhost:4200"
    }
};

// Scopes you add here will be prompted for consent during login
export const loginRequest = {
    scopes: ["https://Promise Ltd.onmicrosoft.com/helloapi/demo.read"] // There shouldn't be any spaces in this string
};

In short, your base authority seems to be incorrect as far as I can tell.

This MSAL and the one with spaces in them I can both sign in an get the jwt.ms token so i suspect the error might be in my b2c configuration, am happy to send you the details

B2C_1_signup_signin_b2c001
B2C_1_reset_passwordb2c001
B2C_1_profile_editingb2c001
Display name: promise-web001

@getwebem please turn on the verbose logs in your MSAL config and share those so we can better figure out what's going on.
// Config object to be passed to Msal on creation
export const msalConfig = {
    auth: {
      ...
    },
    system: {
        loggerOptions: {
            loggerCallback: (level, message, containsPii) => {
                if (containsPii) {
                    return;
                }
                switch (level) {
                    case msal.LogLevel.Error:
                        console.error(message);
                        return;
                    case msal.LogLevel.Info:
                        console.info(message);
                        return;
                    case msal.LogLevel.Verbose:
                        console.debug(message);
                        return;
                    case msal.LogLevel.Warning:
                        console.warn(message);
                        return;
                }
            },
           logLevel: msal.LogLevel.Verbose
        }
    }
};

GET https://promise-web001.b2clogin.com/promise-web001.onmicrosoft.com/b2c_1_signup_signin_b2c001/v2.0/.well-known/openid-configuration 404 (Not Found)

@hectormmg the only  msalConfig file in my code is listed below:
// Config object to be passed to Msal on creation
export const msalConfig = {
auth: {
clientId: "7ab42012-eec1-4e5f-b82d-218d1ec6ce7e",
authority: "https://promise-web001.b2clogin.com/promise-web001.onmicrosoft.com/B2C_1_signup_signin_b2c001",
knownAuthorities: ["promise-web001.b2clogin.com"],
redirectUri: "http://localhost:4200",
postLogoutRedirectUri: "http://localhost:4200"
}
};
// Scopes you add here will be prompted for consent during login
export const loginRequest = {
scopes: ["https://promise-web001.onmicrosoft.com/helloapi/demo.read"]
};
I also double checked the original sample code it doesn't have that code you're refering to:
MSAL.js for React B2C Sample - Authorization Code Flow in Single-Page Applications

@getwebem yes, I'm suggesting you add the system configuration object to your config to turn the MSAL logger on. This way you'll be able to see the logs on the browser console and get more diagnostic info.

This is the code now
// Config object to be passed to Msal on creation
export const msalConfig = {
auth: {
clientId: "7ab42012-eec1-4e5f-b82d-218d1ec6ce7e",
authority: "https://promise-web001.b2clogin.com/promise-web001.onmicrosoft.com/B2C_1_signup_signin_b2c001",
knownAuthorities: ["promise-web001.b2clogin.com"],
redirectUri: "http://localhost:4200",
postLogoutRedirectUri: "http://localhost:4200"
},
system: {
    loggerOptions: {
        loggerCallback: (level, message, containsPii) => {
            if (containsPii) {
                return;
            }
            switch (level) {
                case msal.LogLevel.Error:
                    console.error(message);
                    return;
                case msal.LogLevel.Info:
                    console.info(message);
                    return;
                case msal.LogLevel.Verbose:
                    console.debug(message);
                    return;
                case msal.LogLevel.Warning:
                    console.warn(message);
                    return;
            }
        },
       logLevel: msal.LogLevel.Verbose
    }
}

};
// Scopes you add here will be prompted for consent during login
export const loginRequest = {
scopes: ["https://promise-web001.onmicrosoft.com/helloapi/demo.read"]
};

These are the errors in the console.log()
Uncaught ReferenceError: msal is not defined
at Module. (authConfig.js:34:1)
at Module../src/authConfig.js (authConfig.js:42:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (ProtectedData.jsx:57:1)
at Module../src/ui-components/SignInButton.jsx (SignInButton.jsx:10:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (SignInButton.jsx:10:1)
at Module../src/ui-components/SignInSignOutButton.jsx (SignInSignOutButton.jsx:21:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (Loading.jsx:4:1)
at Module../src/ui-components/NavBar.jsx (NavBar.jsx:29:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (NavBar.jsx:29:1)
at Module../src/ui-components/PageLayout.jsx (PageLayout.jsx:5:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (main.chunk.js:17:83)
at Module../src/App.js (App.js:46:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Module. (authConfig.js:42:1)
at Module../src/index.js (index.js:13:1)
at webpack_require (bootstrap:851:1)
at fn (bootstrap:150:1)
at Object.1 (WelcomeName.jsx:25:1)
at webpack_require (bootstrap:851:1)
at checkDeferredModules (bootstrap:45:1)
at Array.webpackJsonpCallback [as push] (bootstrap:32:1)
at main.chunk.js:1:73
(anonymous) @ authConfig.js:34
./src/authConfig.js @ authConfig.js:42
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ ProtectedData.jsx:57
./src/ui-components/SignInButton.jsx @ SignInButton.jsx:10
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ SignInButton.jsx:10
./src/ui-components/SignInSignOutButton.jsx @ SignInSignOutButton.jsx:21
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ Loading.jsx:4
./src/ui-components/NavBar.jsx @ NavBar.jsx:29
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ NavBar.jsx:29
./src/ui-components/PageLayout.jsx @ PageLayout.jsx:5
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ main.chunk.js:17
./src/App.js @ App.js:46
webpack_require @ bootstrap:851
fn @ bootstrap:150
(anonymous) @ authConfig.js:42
./src/index.js @ index.js:13
webpack_require @ bootstrap:851
fn @ bootstrap:150
1 @ WelcomeName.jsx:25
webpack_require @ bootstrap:851
checkDeferredModules @ bootstrap:45
webpackJsonpCallback @ bootstrap:32
(anonymous) @ main.chunk.js:1
Show 3 more frames
VM70:2 Uncaught ReferenceError: process is not defined
at Object.4043 (:2:13168)
at r (:2:306599)
at Object.8048 (:2:9496)
at r (:2:306599)
at Object.8641 (:2:1379)
at r (:2:306599)
at :2:315627
at :2:324225
at :2:324229
at HTMLIFrameElement.e.onload (index.js:1:1)
4043 @ VM70:2
r @ VM70:2
8048 @ VM70:2
r @ VM70:2
8641 @ VM70:2
r @ VM70:2
(anonymous) @ VM70:2
(anonymous) @ VM70:2
(anonymous) @ VM70:2
e.onload @ index.js:1
be @ index.js:1
(anonymous) @ index.js:1
(anonymous) @ index.js:1
load (async)
be @ index.js:1
(anonymous) @ index.js:1
(anonymous) @ index.js:1
Promise.then (async)
(anonymous) @ index.js:1
(anonymous) @ index.js:1
t @ index.js:1
error (async)
o @ index.js:1
(anonymous) @ index.js:1
ge @ index.js:1
./node_modules/react-dev-utils/webpackHotDevClient.js @ webpackHotDevClient.js:45
webpack_require @ bootstrap:851
fn @ bootstrap:150
1 @ WelcomeName.jsx:25
webpack_require @ bootstrap:851
checkDeferredModules @ bootstrap:45
webpackJsonpCallback @ bootstrap:32
(anonymous) @ main.chunk.js:1
index.js:1 src\authConfig.js
Line 20:26:  'msal' is not defined  no-undef
Line 23:26:  'msal' is not defined  no-undef
Line 26:26:  'msal' is not defined  no-undef
Line 29:26:  'msal' is not defined  no-undef
Line 34:22:  'msal' is not defined  no-undef
Search for the keywords to learn more about each error.

@getwebem add:
import { LogLevel } from "@azure/msal-browser";

And then instead of logLevel: msal.LogLevel.Verbose do logLevel: LogLevel.Verbose

@hectormmg I have attached the updated code as well as the Error results

updated code now

The error from console.log()

@getwebem could you file a support ticket with the B2C service team? This is most likely related to your B2C configuration and not MSAL itself, thanks!


feat!: move evidence back to Tendermint block

Closes: https://github.com/celestiaorg/celestia-core/issues/894
The CI check proto-breakage is expected to fail (here) because this PR makes breaking changes to the protobuf definitions.

There is an issue with this test: https://github.com/rootulp/celestia-core/blob/2656054788d3923edf4839376d5a97f9c3a707ab/node/node_test.go#L225 which calls https://github.com/rootulp/celestia-core/blob/2656054788d3923edf4839376d5a97f9c3a707ab/node/node_test.go#L225. It looks like the amount of data in the proposal block exceeds maxDataBytes. This may be resolved by updating the calculation used to determine maxDataBytes.

This looks like it was resolved after accounting for the varint plus ceasing to double the data, no?

Yep, resolved by reverting the celestia-core modifications to the mempool back to vanilla Tendermint in https://github.com/celestiaorg/celestia-core/pull/897/commits/a158519e2ec063d19a7fd9a6c30f2462a0683f71 and https://github.com/celestiaorg/celestia-core/pull/897/commits/34fa0fd7fce79f529c588ee30ced63df5a423d00. I just marked the comment as resolved.

LGMT, we might need a tag for super-mega-breaking.

Agreed. I just created and added the label "proto breaking". I infer that a celestia-app release with this change won't be compatible with existing testnets so we may need a strategy meeting on how to upgrade all testnets after we cut a celestia-app release with this change.


stream error

Connection org.mariadb.jdbc.MySQLConnection@5aa2b592 failed alive test with exception Could not read resultset: unexpected end of stream, read 0 bytes from 4
version: 2.4.1
pool.connectionTimeout=3000
pool.validationTimeout=1000
pool.idleTimeout=90000
pool.maxLifetime=123000
pool.minimumIdle=400
pool.initializationFailFast=false
mvn.pool.leakDetectionThreshold=10000

minimumIdle value looks suspicious. probably mysql or the machine is not capable/configured to handle such load. If you missed, https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing may help you decide pool size.
Please also upgrade to release 2.4.3

Agreed. Unless you have 200+ CPU cores, 400 connections is insane.  This error looks like an out of memory error in MySQL.  Either way, the origin of this error is not HikariCP.

OK

Our use case is not OLTP and normal scenario. My application needs more connections to kill query of OLTP , that slow-query.

Agreed. Unless you have 200+ CPU cores, 400 connections is insane.

EDIT: check and if required set "wait_time" in mysql to minute or two more than maxLifetime

@brettwooldridge
idleTimeout > maxLifetime is right ?
or
idleTimeout < maxLifetime is right ?

idleTimeout should be less than maxLifetime.

@brettwooldridge  Thanks


Send a helpful message to help sessions that go unanswered

Around 23% of help channel sessions go completely unanswered.
We have a theory that a large proportion of this stat are from people who don't ask their question well, or don't provide enough information for helpers to help them.
To try and reduce that, this issue proposes that we send a message in help channels after 15 minutes if no one other than the claimant has sent a message.
I believe this message should include:

An explanation that everyone here is a volunteer, and we don't have 24/7 cover, so asking your question at another time may get better results
Reiterate the more important points from the available help channel embed.

We should also make sure that the same person doesn't get this message multiple times, since it could become infuriating if you receive this embed every time you ask a question through no fault of your own.
To aid this, we should store claimants who receive this embed in redis, for ease of implementation.

An addition to this I thought of is when someone claims a how-to-get-help channel it could check if their initial message is in a list of common first messages like hello, hi, help me etc. and if it does have the bot wait say 10 seconds and if the user that claimed the channel is not typing (I believe you can check that, if you can't then have this happen after say 60 seconds of no messages) have the bot reply to the message restating:
Remember to:
• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you got one.
• Explain what you expect to happen and what actually happens.

Another thing I also think it should say is something a long the lines of:
"Try to write your sentences with grammar heavily in mind as it will help readability for people trying to answer your question"

To aid this, we should store claimants who receive this embed in redis, for ease of implementation.

Instead of using redis, this should be stored in the database as redis is no longer being used for anything persistent.

This was implemented in https://github.com/python-discord/bot/pull/2504


FIX THE FOOTER

FIX the FOOTER
INFO:
Make the footer look more presentable
SKILLS NEEDED:
BASIC HTML, INTERMEDIATE CSS, AND (OPTIONAL) JAVASCRIPT
RULES:

Follow our coding style, don't incorporate your own (not talking about design style here, I'm talking about syntax)
Make the footer element look better

PR submitted


double-beep: Blacklist Kishankant\s+Yadav

double-beep requests the blacklist of the username Kishankant\s+Yadav. See the MS search here and the Stack Exchange search here.

Kishankant\s+Yadav has been seen in 8 true positives, 0 false positives, and 0 NAAs.

!!/approve

Approved by tripleee in Charcoal HQ


Update Main.java

Calls stop on currentTest (ApplicationListener) allows clean exit (call dispose) from currrently running ApplicationListener

Thanks!

Your welcome.
You are my first pull request merge. So than you.
Also thank you for the blog examples and thank you for libgdx.
Regards
Mark


Global snippets: group multiple file types, snippet inheritance

Grouping global snippets by file type in the same way settings can be grouped.
"[js][ts]":{
	.... snippets
}

And/Or better (method 2):
// typescript.json  (the current snippet file)
"inherit": ["javascript"]   // <- this is the key line
.... snippets

The "scope" field can be useful in certain special cases but usually snippets come in groups, so that is not enough.
Currently if you have 100 js snippets and want them in ts files you'd have to edit the "scope" field of each one of them. Then you're trying React - do it again.... And so on.
The 2 methods are similar and can complement each other but the advantage of method 2 is that a more specialized version of a language can add additional snippets and override inherited snippets, eg for(var i: number; ... overrides for(var i; ... if the for snippet is redefined, but doesn't have to.
While we're at it, the typescript.json file can contain a "exclude javascript": [...snippet ids] field naming inherited snippets that you do not want in a ts file.

Current VSCode snippet configuration is not flex enough, so many snippet extension programmatically to generator snippets. Like https://github.com/dsznajder/vscode-react-javascript-snippets and https://github.com/johnpapa/vscode-angular-snippets which user ammount is vary large.


Add initial VMVX matrix intrinsics.

As of the initial work in #5536 all operations are expanded to scalar math. This is the most general path but also the slowest. To get VMVX running fast enough to be usable we need to take the top N% of ops consuming time and turn them into intrinsics. Thankfully we know what these are (in BERT at least): vector reduction and matrix transpose and multiply.
The skeleton is in-place for adding these intrinsics but work is required on the compiler side to define them in VMVXOps.td, convert to them with VectorToVMVX patterns, and lower into the vmvx.imports.mlir imports. At runtime the exports.inl and vmvx_module.c would get their implementation.

I'm interested in hacking around on this. For reference, I was able to inject vmvx.addi32 and vmvx.addi64 ops and intrinsics end-to-end into the pipeline, so I know the overall structure now. If this is off the immediate critical path, I'd love to take a crack at it.
Do you have any high-level thoughts on what we want to do here? Create some vmvx.vector ops and them implement them with some vector SIMD instructions in module.c? I assume we will want to convert the vector ops into more machine-like sets of operations for the intrinsics? At the least, I can go hack around for a little bit and come back with more questions etc.

That'd be fantastic!
I think starting with something that has a big performance benefit and some prior uses in other lowering paths like vector.contract would let you focus on what unique aspects there are here. It's also a fun one to optimize with intrinsics.
As an example starting point we have a vector.contract -> SPIR-V cooperative matrix lowering here that would likely look similar in approach:
https://github.com/google/iree/blob/74e33026bc9972e31ee793205fdfada78f67fecd/iree/compiler/Conversion/LinalgToSPIRV/test/vector_to_cooperative_matrix.mlir
The lowering into the vector.contract could be taken from this path and then the patterns exist as templates to identify this situation and can be swapped with one that targets vmvx. The definition will differ in vmvx as it'll likely be a single op that takes the stride information/etc vs having the split load/matmul/store, but conceptually it'll be the same (some lowering into vector dialect and then identification of the contract and pulling in the reads/writes to buffers + offsets to pass over the fence to module.c).
Getting vector.contract alone working would probably yield a 100x speedup (at least - I'd wager closer to 1000x) - which would feel pretty good :)

Thanks for the guidance. I started looking at that SPIR-V lowering, and it's a great starting point. Gonna hack for a bit and come back / post in discord.
100x-1000x would feel awesome :)

@benvanik so it took me a little bit of time to parse SPIRV lowering logic in LinalgToSPIRV/TileAndVectorizineInOneWorkgroupPass.cpp but I am starting to follow along at a high level. It looks like a lot of that code is tied in to the SPIRV TargetEnv (initGPULaunchConfig etc). I think reworking this code to work for VMVX will take some time --- which I hope to do --- but I think getting a quick end to end might make it easier to progressively work on this.
For example, from grabbing a little bit of the vectorization logic out of TileAndVectorizeInOneWorkgroupPass, and placing it at the start of buildVectorVMVXTransformPassPipeline, I can get some vectorization.
func @elem_add(%arg0 : tensor<4096xi32>, %arg1 : tensor<4096xi32>) -> tensor<4096xi32> attributes { iree.module.export } {
  %0 = "mhlo.add"(%arg0, %arg1) : (tensor<4096xi32>, tensor<4096xi32>) -> tensor<4096xi32>
  return %0 : tensor<4096xi32>
}

func @matmul(%arg0 : tensor<4096x4096xf16>, %arg1 : tensor<4096x4096xf16>) -> tensor<4096x4096xf16> attributes { iree.module.export } {
  %0 = "mhlo.dot"(%arg0, %arg1) : (tensor<4096x4096xf16>, tensor<4096x4096xf16>) -> tensor<4096x4096xf16>
  return %0 : tensor<4096x4096xf16>
}

will get me to
func @elem_add_dispatch_0() {
  %c4096 = constant 4096 : index
  %c0_i32 = constant 0 : i32
  %c0 = constant 0 : index
  %0 = hal.interface.binding.subspan @io::@s0b0_ro_external[%c0] : memref<4096xi32>
  %1 = hal.interface.binding.subspan @io::@s0b1_ro_external[%c0] : memref<4096xi32>
  %2 = hal.interface.binding.subspan @io::@s0b2_xw_external[%c0] : memref<4096xi32>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 128)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 128)>()[%workgroup_count_x]
  scf.for %arg0 = %3 to %c4096 step %4 {
    %5 = memref.subview %0[%arg0] [128] [1] : memref<4096xi32> to memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>
    %6 = memref.subview %1[%arg0] [128] [1] : memref<4096xi32> to memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>
    %7 = memref.subview %2[%arg0] [128] [1] : memref<4096xi32> to memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>
    %8 = vector.transfer_read %5[%c0], %c0_i32 {in_bounds = [true]} : memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>, vector<128xi32>
    %9 = vector.transfer_read %6[%c0], %c0_i32 {in_bounds = [true]} : memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>, vector<128xi32>
    %10 = addi %8, %9 : vector<128xi32>
    vector.transfer_write %10, %7[%c0] {in_bounds = [true]} : vector<128xi32>, memref<128xi32, affine_map<(d0)[s0] -> (d0 + s0)>>
  }
  return
}

func @matmul_dispatch_0() {
  %c4096 = constant 4096 : index
  %c64 = constant 64 : index
  %c4 = constant 4 : index
  %c32 = constant 32 : index
  %cst = constant dense<0.000000e+00> : vector<64x64xf16>
  %cst_0 = constant 0.000000e+00 : f16
  %cst_1 = constant dense<0.000000e+00> : vector<4x4xf16>
  %c0 = constant 0 : index
  %0 = hal.interface.binding.subspan @io::@s0b0_ro_external[%c0] : memref<4096x4096xf16>
  %1 = hal.interface.binding.subspan @io::@s0b1_ro_external[%c0] : memref<4096x4096xf16>
  %2 = hal.interface.binding.subspan @io::@s0b2_xw_external[%c0] : memref<4096x4096xf16>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 64)>()[%workgroup_id_y]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 64)>()[%workgroup_count_y]
  scf.for %arg0 = %3 to %c4096 step %4 {
    %5 = affine.apply affine_map<()[s0] -> (s0 * 64)>()[%workgroup_id_x]
    %6 = affine.apply affine_map<()[s0] -> (s0 * 64)>()[%workgroup_count_x]
    scf.for %arg1 = %5 to %c4096 step %6 {
      %7 = memref.subview %0[%arg0, 0] [64, 4096] [1, 1] : memref<4096x4096xf16> to memref<64x4096xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
      %8 = memref.subview %1[0, %arg1] [4096, 64] [1, 1] : memref<4096x4096xf16> to memref<4096x64xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
      %9 = memref.subview %2[%arg0, %arg1] [64, 64] [1, 1] : memref<4096x4096xf16> to memref<64x64xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
      vector.transfer_write %cst, %9[%c0, %c0] {in_bounds = [true, true]} : vector<64x64xf16>, memref<64x64xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
      scf.for %arg2 = %c0 to %c64 step %c32 {
        scf.for %arg3 = %c0 to %c64 step %c32 {
          scf.for %arg4 = %c0 to %c4096 step %c32 {
            %10 = memref.subview %7[%arg2, %arg4] [32, 32] [1, 1] : memref<64x4096xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
            %11 = memref.subview %8[%arg4, %arg3] [32, 32] [1, 1] : memref<4096x64xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
            %12 = memref.subview %9[%arg2, %arg3] [32, 32] [1, 1] : memref<64x64xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
            scf.for %arg5 = %c0 to %c32 step %c4 {
              scf.for %arg6 = %c0 to %c32 step %c4 {
                scf.for %arg7 = %c0 to %c32 step %c4 {
                  %13 = memref.subview %10[%arg5, %arg7] [4, 4] [1, 1] : memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
                  %14 = memref.subview %11[%arg7, %arg6] [4, 4] [1, 1] : memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
                  %15 = memref.subview %12[%arg5, %arg6] [4, 4] [1, 1] : memref<32x32xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>> to memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
                  %16 = vector.transfer_read %13[%c0, %c0], %cst_0 {in_bounds = [true, true]} : memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>, vector<4x4xf16>                                                                                %17 = vector.transfer_read %14[%c0, %c0], %cst_0 {in_bounds = [true, true], permutation_map = affine_map<(d0, d1) -> (d1, d0)>} : memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>, vector<4x4xf16>
                  %18 = vector.transfer_read %15[%c0, %c0], %cst_0 {in_bounds = [true, true]} : memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>, vector<4x4xf16>
                  %19 = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>, affine_map<(d0, d1, d2) -> (d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1)>], iterator_types = ["parallel", "parallel", "reduction"], kind = #vector.kind<add>} %16, %17, %cst_1 : vector<4x4xf16>, vector<4x4xf16> into vector<4x4xf16>
                  %20 = addf %18, %19 : vector<4x4xf16>
                  vector.transfer_write %20, %15[%c0, %c0] {in_bounds = [true, true]} : vector<4x4xf16>, memref<4x4xf16, affine_map<(d0, d1)[s0] -> (d0 * 4096 + s0 + d1)>>
                }
              }
            }
          }
        }
      }
    }
  }
  return
}

which gets me an elem wise vector add and a contract to begin writing the conversion to VMVX vector ops (per some of the discussion above and https://github.com/google/iree/issues/5549) and some initial intrinsics. After that, I can loop back to this lowering and start building up the logic there.

@anthonycanino1 nice! there are a few approaches to how to handle vectors in VMVX (we can discuss on #5549 - which is out of date and I should update - or discord, whatever works for you!), but building an understanding about the tiling approaches like you are is the first step (and you may soon know more than I do about that part of the code :).
As you can see the levels of tiling and the sizes involved can vary quite a bit and we'll likely have entirely unique configurations for VMVX to try to hide the overhead involved with making calls out to the native implementations (doing 4 elements is no good, 4096 is better, and if it's worth going beyond that we'll have to measure).
I think the big decision point is what to do about vector. My instinct is that allocating them as memory will be the easiest and most flexible path. Will dump some notes in #5549.

Closing for now in favor of using #5549 to track the vectorization/intrinsics work.


[WIP] [flat] N64 Overlay: Independent C-Button support

Added set of overlay pages to properly address the ParaLLEl "Independent C-Button" option, and added a button to switch between the two overlay groups.

Hi, good work :) Does this affect the loading speed at all? If so, we might want to break it off to its own overlay.

Oh dear. I just tested it. Everything's screwy. To the drawing board again. I'll be back.

Okay, so all my button inputs match up properly to those specified in the ParaLLEl core under alternate input, but they act as if I had pressed the specified inputs on a normal controller, e.g. c-left has the function of the left trigger and c-down has the function of A, instead of both acting as, well, c-buttons.
Am I misunderstanding the effects of independent c-button mode?

Hmm, that's a good question. The overlays act strangely sometimes insofar as I don't think they're affected by the input remapping(?), so maybe they sidestep the independent c-button mode, too?
But I guess that can't be right because you said the independent c-button option screwed up the normal overlay for you, right?

Yep. I would presume so. :/
However (!), after a good deal of testing, I think I may have solved it. I'll get back to you after dinner.

Alright. I figured out what I did wrong, and I made a new overlay with the n64.cfg from gamepads/n64 .
I have yet to re-revamp gamepads/flat/nintendo64.cfg, although that is my next step. I'll open up the other pull request, and mark this one WIP until I get around to it, I suppose.

Looks good with the regular overlay. You're still working on the flat version?

I'm going to close this pull request and try this again; I don't fully understand GitHub.


Update meeting details/minutes info

Clean up the meeting section; update to proper Zoom info and note that
it is currently weekly. Add section to clean up where minutes are found
and the use of HackMD currently.
Closes: #10, #11, #13
// cc: @vbatts
Signed-off-by: Phil Estes<EMAIL_ADDRESS>

LGTM