Improve taint propagation analysis for 'check' expressions

Purpose
$subject
As check expression may cause a function to return with provided error, we need to consider check expressions when calculating the return value taintedness of a function.
public function main(string... args) {
    error? k = foo();
}

function foo() returns error? {
    _ = check bar(); // check may return the error, which is tainted
}

function bar() returns @tainted string|error {
    return error("reason", message = "tainted data here");
}

Fixes https://github.com/ballerina-platform/ballerina-lang/issues/19610
Approach

Describe how you are implementing the solutions along with the design details.

Samples

Provide high-level details about the samples related to this feature.

Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

Check List

[ ] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #19630 into ballerina-1.0.x will not change coverage.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           ballerina-1.0.x   #19630   +/-   ##
================================================
  Coverage            20.51%   20.51%
================================================
  Files                   52       52
  Lines                 1394     1394
  Branches               216      216
================================================
  Hits                   286      286
  Misses                1095     1095
  Partials                13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1012940...a6ae83c. Read the comment docs.

@wggihan Pls review changes.


help me

please tell me what to do:java.lang.NullPointerException

More information would be helpful. Can you upload the log file (to Pastebin) and post the link here, please?
Maybe you can find help in the forum,too.

Also helps to know what computer you're playing on. If it is really old it probably won't run Terasology :(
If you got the CrashReporter window then there is more information on there. Even if that's gone now if you're on Windows you should be able to find the log again at C:\Users[username]\Saved Games\Terasology\logs

Closing from lack of user feedback


Sarch 06 docker

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

9 of 41 (21.95%)  changed or added relevant lines in 14 files are covered.
7 unchanged lines in 4 files lost coverage.
Overall coverage increased (+34.6%) to 61.086%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/DiceCupService.scala
0
2
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/FieldService.scala
0
2
0.0%

fileio/src/main/scala/de/htwg/sa/kniffel/fileio/FileIOService.scala
0
2
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/GameService.scala
0
2
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/api/FieldApi.scala
0
3
0.0%

fileio/src/main/scala/de/htwg/sa/kniffel/fileio/api/FileIoApi.scala
0
3
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/api/GameApi.scala
0
3
0.0%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/api/DiceCupApi.scala
0
4
0.0%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/RestControllerService.scala
0
4
0.0%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/api/RestControllerApi.scala
0
7
0.0%

Files with Coverage Reduction
New Missed Lines
%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/api/RestControllerApi.scala
1
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/api/FieldApi.scala
2
0.0%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/api/DiceCupApi.scala
2
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/api/GameApi.scala
2
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
34.6%

Covered Lines:
540

Relevant Lines:
884

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

9 of 41 (21.95%)  changed or added relevant lines in 14 files are covered.
7 unchanged lines in 4 files lost coverage.
Overall coverage increased (+34.6%) to 61.086%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/DiceCupService.scala
0
2
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/FieldService.scala
0
2
0.0%

fileio/src/main/scala/de/htwg/sa/kniffel/fileio/FileIOService.scala
0
2
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/GameService.scala
0
2
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/api/FieldApi.scala
0
3
0.0%

fileio/src/main/scala/de/htwg/sa/kniffel/fileio/api/FileIoApi.scala
0
3
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/api/GameApi.scala
0
3
0.0%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/api/DiceCupApi.scala
0
4
0.0%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/RestControllerService.scala
0
4
0.0%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/api/RestControllerApi.scala
0
7
0.0%

Files with Coverage Reduction
New Missed Lines
%

restcontroller/src/main/scala/de/htwg/sa/kniffel/controller/api/RestControllerApi.scala
1
0.0%

field/src/main/scala/de/htwg/sa/kniffel/field/api/FieldApi.scala
2
0.0%

dicecup/src/main/scala/de/htwg/sa/kniffel/dicecup/api/DiceCupApi.scala
2
0.0%

game/src/main/scala/de/htwg/sa/kniffel/game/api/GameApi.scala
2
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
34.6%

Covered Lines:
540

Relevant Lines:
884

💛  - Coveralls


AbstractCollocationFinder from_documents() issues

If we pass a set of documents (e.g. tweets) to the from_documents() method we also get bigrams that never appeared in any document. This is due to _itertools.chain(*documents), which collapses all the documents (so we end up with a single big document and we extract its bigrams).
def from_documents(cls, documents):
    """Constructs a collocation finder given a collection of documents,
    each of which is a list (or iterable) of tokens.
    """
    return cls.from_words(_itertools.chain(*documents))

For example,using a list of two documents:
docs = [['I','am','a','farmer'], ['Newspapers','are','interesting']]

finder = BigramCollocationFinder.from_documents(docs)
finder.nbest(BigramAssocMeasures.raw_freq, 10)

>>> [('I', 'am'), ('Newspapers', 'are'), ('a', 'farmer'), ('am', 'a'), ('are', 'interesting'), ('farmer', 'Newspapers')]

So the method will also generate the bigram ('farmer', 'Newspapers'), which actually never occurs in the documents.
Con we consider it as an issue?

Yes, it's an issue. A simple solution may involve putting n placeholders
between documents.
On 3 July 2015 at 18:40, Pierpaolo Pantone<EMAIL_ADDRESS>wrote:

If we pass a set of documents (e.g. tweets) to the from_documents()
method we also get bigrams that never appeared in any document. This is due
to _itertools.chain(*documents), which collapses all the documents (so we
end up with a single big document and we extract its bigrams).
def from_documents(cls, documents):
"""Constructs a collocation finder given a collection of documents,
each of which is a list (or iterable) of tokens.
"""
return cls.from_words(_itertools.chain(*documents))
For example,using a list of two documents:
docs = [['I','am','a','farmer'], ['Newspapers','are','interesting']]
finder = BigramCollocationFinder.from_documents(docs)
finder.nbest(BigramAssocMeasures.raw_freq, 10)

[('I', 'am'), ('Newspapers', 'are'), ('a', 'farmer'), ('am', 'a'), ('are', 'interesting'), ('farmer', 'Newspapers')]

So the method will also generate the bigram ('farmer', 'Newspapers'),
which actually never occurs in the documents.
Con we consider it as an issue?
—
Reply to this email directly or view it on GitHub
https://github.com/nltk/nltk/issues/1033.

@longdt219 would you please implement @jnothman's suggestion.

Sure @stevenbird,

@fievelk is this satisfactorily resolved now?

I can confirm that the issue seems to be solved now:
>>> docs = [['I','am','a','farmer'], ['Newspapers','are','interesting']]

>>> finder = BigramCollocationFinder.from_documents(docs)
>>> finder.nbest(BigramAssocMeasures.raw_freq, 10)

[('I', 'am'), ('Newspapers', 'are'), ('a', 'farmer'), ('am', 'a'), ('are', 'interesting')]

The bigram ('farmer', 'Newspapers') is now correctly omitted.

Resolved in #1086


Add linear analysis to Makefile+README; implement Annas loop suggestion

Are you referring to the actc.txt file? If so, I can put it anywhere you want. I just thought you meant you wanted it in code/scripts.

Good. code/scripts is exactly where I want it go.


Handle context for :up, :bound

I'm not exactly sure what causes this in my environment, but i recently started hitting this:
** (FunctionClauseError) no function clause matching in Nerves.InterimWiFi.WiFiManager.consume/3
  (nerves_interim_wifi) lib/interim_wifi/wifi_manager.ex:149: consume(:up, {:bound,
  %{ domain: "T-mobile.com",
       ifname: "wlan0",
       ipv4_address: "<IP_ADDRESS>",
       ipv4_broadcast: "<IP_ADDRESS>",
       ipv4_gateway: "<IP_ADDRESS>",
       ipv4_subnet_mask: "<IP_ADDRESS>",
      nameservers: ["<IP_ADDRESS>"]
  }
}, %Nerves.InterimWiFi.WiFiManager{
     context: :up,
     dhcp_pid: #PID<0.496.0>,
     ifname: "wlan0",
     settings: [ssid: "supersecretssid", key_mgmt: :"WPA-PSK", psk: "super_secret_password"],
     wpa_pid: #PID<0.414.0>
})

Not entirely sure if this is the correct fix.

I think :noop'ing this message is okay @fhunleth any thoughts? looks good to me

Yes - it seems like the right move is to ignore it. I assume the connection is bouncing on the initial connect. The way I read it is that you're getting a DHCP response before you expect it, so I assume it's from a previous connection that bounced when the DHCP response came in.

I believe that is what is happening @fhunleth


Texture: add support for N-channel textures

With this PR enoki::Texture can now handle textures with an arbitrary number of channels.
All evaluation interfaces were changed, they now take an output parameter that is used to write the evaluation's result to. The python bindings for the evaluation methods still directly return the result, however these are now returned as python list of length N for a N-channel texture.
Tests were added to cover the new interfaces and verify the underlying implementation in enoki-jit when using CUDA texture objects.
:warning: This PR is already rebased on Ziyi's work in https://github.com/wjakob/enoki/pull/23 (not merged yet).

@njroussel, I merged the Enoki-JIT part. Does this need adaptations? (e.g. to point to the updated Enoki-JIT repo)

Yup, I updated the submodule. Ready to go now :rocket:

Excellent, thank you Nicolas!


Move StatusCode toString to StatusCode

It now lives in Response

unfortunately __toString() is not possible with current versions of PHP.
But a method is possible


Can't login after scan the qrcode

Just got the token begin with "puppet_donut_*******". I tested with the code below. I can login at the first time(the message bar in my mobile wechat showes already login), but when I logged out. I can't login anymore, no logs and no message bar shows after I scan the qrcode, how can I resolve this issue? Thanks
const token = "puppet_donut_***********"

const bot = new Wechaty({
  puppet: 'wechaty-puppet-hostie',
  puppetOptions: {
    token,
  }
});

bot
  .on('scan', (qrcode, status) => {
    if (status === ScanStatus.Waiting) {
      QrcodeTerminal.generate(qrcode, {
        small: true
      })
    }
  })
  .on('login', async user => {
    console.log(`user: ${JSON.stringify(user)}`)
  })
  .on('message', async message => {
    console.log(`message: ${JSON.stringify(message)}`)
  })
  .start()

我也一样的问题，退出后一直无法在登录了，二维码也不见显示。
开了 WECHATY_LOG=silly，运行日志如下：
C:\Users\Administrator>node C:\nodejs\wechaty\index.js
16:57:50 SILL Puppet Config: WECHATY_LOG set level to silly
16:57:51 INFO Config registering process.on("unhandledRejection") for developmen
t/debug
16:57:51 VERB Config constructor()
16:57:51 VERB Wechaty constructor()
16:57:51 VERB StateSwitch constructor(Wechaty, "{"log":{"enableTimestamp":true,"
logLevel":5,"prefixFilter":{}}}")
16:57:51 VERB StateSwitch constructor(WechatyReady, "{"log":{"enableTimestamp":t
rue,"logLevel":5,"prefixFilter":{}}}")
16:57:51 VERB Wechaty on(scan, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty on(login, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty on(friendship, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty on(message, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty on(room-invite, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty on(room-join, listener) registering... listenerCount: 0
16:57:51 VERB Wechaty () start() v0.51.4 is starting...
16:57:51 VERB Wechaty id: cki01966f00003wkgeuqpcful
16:57:51 SILL StateSwitch  on() is false
16:57:51 SILL StateSwitch  off() is true
16:57:51 VERB StateSwitch  off(true) <- (true)
16:57:51 SILL StateSwitch  on() is false
16:57:51 VERB StateSwitch  on(pending) <- (false)
16:57:51 VERB Wechaty initPuppet() wechaty-puppet-hostie
16:57:51 VERB PuppetManager resolve({puppet: wechaty-puppet-hostie, puppetOption
s: {"token":"puppet_donut_xxxxxxx"}})
16:57:51 VERB PuppetManager resolveName(wechaty-puppet-hostie)
16:57:51 VERB PuppetManager checkModule(wechaty-puppet-hostie)
16:57:51 SILL PuppetManager checkModule() wechaty-puppet-hostie installed versio
n 0.11.10 satisfied range *
16:57:51 VERB Puppet constructor({"token":"puppet_donut_xxxxxxxxxx"}) #0
16:57:51 VERB StateSwitch constructor(PuppetHostie, "{"log":{"enableTimestamp":t
rue,"logLevel":5,"prefixFilter":{}}}")
16:57:51 VERB Puppet constructor() watchdog timeout set to 60 seconds
16:57:51 VERB Puppet constructor() childClassPath=C:\nodejs\wechaty\node_modules
\wechaty-puppet-hostie\dist\src\client
16:57:51 VERB Puppet setMemory()
16:57:51 VERB Wechaty initPuppetEventBridge(Puppet#0())
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(friendship) (listenerCou
nt:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(login) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(logout) (listenerCount:0
) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(message) (listenerCount:
0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(room-invite) (listenerCo
unt:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(room-join) (listenerCoun
t:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(room-leave) (listenerCou
nt:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(room-topic) (listenerCou
nt:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(scan) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(dirty) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(dong) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(error) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(heartbeat) (listenerCoun
t:0) registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(ready) (listenerCount:0)
registering...
16:57:51 VERB Wechaty initPuppetEventBridge() puppet.on(reset) (listenerCount:0)
registering...
16:57:51 VERB Wechaty wechatifyUserModules(Puppet#0())
16:57:51 VERB PuppetHostie start()
16:57:51 SILL StateSwitch  on() is false
16:57:51 SILL StateSwitch  on() is false
16:57:51 VERB StateSwitch  on(pending) <- (false)
16:57:51 VERB PuppetHostie startGrpcClient()
16:57:51 VERB PuppetHostie discoverHostieIp(puppet_donut_xxxxxxxxx)
16:57:52 VERB PuppetHostie startGrpcStream()
16:57:52 SILL StateSwitch  on() is pending
16:57:52 VERB StateSwitch  on(true) <- (pending)
16:57:52 VERB Wechaty on(heartbeat, listener) registering... listenerCount: 0
16:57:52 SILL StateSwitch  on() is pending
16:57:52 VERB StateSwitch  on(true) <- (pending)
16:57:52 VERB Wechaty  stop() v0.51.4 is stopping ...
16:57:52 SILL StateSwitch  off() is false
16:57:52 SILL StateSwitch  off() is true
16:57:52 VERB StateSwitch  off(true) <- (true)
16:57:52 SILL StateSwitch  off() is false
16:57:52 VERB StateSwitch  off(pending) <- (false)
16:57:52 VERB PuppetHostie stop()
16:57:52 SILL StateSwitch  off() is false
16:57:52 SILL StateSwitch  off() is false
16:57:52 VERB StateSwitch  off(pending) <- (false)
16:57:52 VERB PuppetHostie stopGrpcStream()
16:57:52 VERB PuppetHostie startGrpcStream() eventStream.on(end)
16:57:52 VERB PuppetHostie stopGrpcClient()
16:57:52 SILL StateSwitch  off() is pending
16:57:52 VERB StateSwitch  off(true) <- (pending)
16:57:52 SILL StateSwitch  off() is pending
16:57:52 VERB StateSwitch  off(true) <- (pending)
C:\Users\Administrator>

Hi all, I got the same issue as well.
One more thing, after I confirm login in my iPhone screen, it can kick out my Mac Wechat which means the QRCode is fine.

"I can't login anymore, no logs and no message bar shows after I scan the qrcode"
I think we actually logon success from Wechat service perspective,
@huan Could you please help to check why no logs ( suppose to log the 'login') and even no 'message' event logs.

Here's the bot creation I copy from a sample shared by juzibot.
// not assign attribute 'name' as I'm about to use two wechat account for testing separately.
const bot = new Wechaty({
puppet: 'wechaty-puppet-hostie',
puppetOptions: {
token_which_expired_in_15_days,
}
});
// dependencies
"dependencies": {
"qrcode-terminal": "^0.12.0",
"wechaty": "^0.51.10",
"wechaty-puppet-hostie": "^0.10.8"
},
if any kind of screenshot needed, please ping me, many thx.

I guess this is a donut related bug.
Could you please try to use wechaty-puppet-puppeteer to log in your WeChat account?
If you can get any response after you scan the qr code, then the Wechaty is good, and then you should contact the donut service provider for support.

@huan
Hi, thanks for your quick response and it took me a long time to install that puppeteer.
Yes , i got the 'To protect your account, logging in to WeChat via the web has been suspended ...' message after log in.
BTW, could you please share the contact point of donut service support or raise a ticket for us ? Coz I'm new here :)

@KochamCie You are welcome!
Please go to https://github.com/wechaty/puppet-services to file puppet service-related issues.

Please feel free to ask for support from our community at https://gitter.im/wechaty/wechaty

Please feel free to ask for support from our community at https://gitter.im/wechaty/wechaty


HDF5 DataIter Python

I basically put together a DataIter for HDF5 data in python. It is pretty much a clone of NDArrayDataIter  (without shuffeling as it is out of core).
See code and example below. The idea is to load data into memory batch by batch, hence it performs slower than NDArrayIter which has everything in memory.
(getlabel() update 15.08.2017)
class HDF5ArrayIter(DataIter):

    def __init__(self, data, label, batch_size=1,
                 data_name="data", label_name="softmax_label",
                 last_batch_handle="pad"):
        super(HDF5ArrayIter, self).__init__(batch_size)

        self.data = data
        self.label = label
        self.batch_size = batch_size
        self.cursor = -batch_size
        self.num_data = data.shape[0]
        self.last_batch_handle = last_batch_handle
        self.data_name = data_name
        self.label_name = label_name

    @property
    def provide_data(self):
        return [DataDesc(self.data_name,
                         tuple([self.batch_size] + list(self.data.shape[1:])),
                         self.data.dtype)]

    @property
    def provide_label(self):
        return [DataDesc(self.label_name,
                         tuple([self.batch_size] + list(self.label.shape[1:])),
                         self.label.dtype)]

    def hard_reset(self):
        self.cursor = -self.batch_size

    def reset(self):
        if self.last_batch_handle == "roll_over" and self.cursor > self.num_data:
            self.cursor = -self.batch_size + (self.cursor % self.num_data) % self.batch_size
        else:
            self.cursor = -self.batch_size

    def iter_next(self):
        self.cursor += self.batch_size
        return self.cursor < self.num_data

    def next(self):
        if self.iter_next():
            return DataBatch(data=self.getdata(),
                             label=self.getlabel(),
                             pad=self.getpad(),
                             index=None)
        else:
            raise StopIteration

    def getdata(self):
        assert(self.cursor < self.num_data), "DataIter needs reset."
        if self.cursor + self.batch_size <= self.num_data:
            return [array(self.data[self.cursor:self.cursor+self.batch_size])]
        else:
            pad = self.batch_size - self.num_data + self.cursor
            return [concatenate(array(self.data[self.cursor:]), array(self.data[:pad]))]

    def getlabel(self):
        assert(self.cursor < self.num_data), "DataIter needs reset."
        if self.cursor + self.batch_size <= self.num_data:
            return [array(self.label[self.cursor:self.cursor+self.batch_size])]
        else:
            pad = self.batch_size - self.num_data + self.cursor
            return [concatenate(array(self.label[self.cursor:]), array(self.label[:pad]))]

    def getpad(self):
        if self.last_batch_handle == "pad" and self.cursor + self.batch_size > self.num_data:
            return self.cursor + self.batch_size - self.num_data
        else:
            return 0

And finally to be used as follows:
with h5py.File("myDataFile.hdf5", "r") as h5:
    X = h5["myData"]
    y = h5["myLabels"]
    train_iter = HDF5ArrayIter(X, y, BATCH_SIZE, last_batch_handle="pad")

I can submit a PR and add docs if people find it a good enough idea.
Comments?
Cheers
Andre

I use this code but have error.What library should be import?

But no example anout how to use oout-of-training for hdf5

Here a toy example where I first write the data to disk, then load it back (assuming this would be a different session...) and iterate over it with NDArrayIter
import numpy as np
import h5py
import mxnet as mx

N = 1000

X = np.random.normal(0,1,(N,12))
y = np.random.randint(0,2,N)

# write data to file
with h5py.File('myfile.hdf5', "w") as ofile:
    ofile.create_dataset("X", data=X)
    ofile.create_dataset("y", data=y)

# load data from file
ifile = h5py.File('myfile.hdf5', 'r')

X_h5 = ifile["X"]
y_h5 = ifile["y"]

batch_size = 200
dataiter = mx.io.NDArrayIter(X_h5, y_h5, batch_size=batch_size)

for iBatch, batch in enumerate(dataiter):
    print(iBatch, batch.data[0].asnumpy().shape)

ifile.close()

wow,Thank you so much.But sometime I want to train a RNN,and it has varing time-steps,can it be load as hdf5 format then uising NDArrayIter?

Sorry I have never done anything with RNNs, so I dont know the answer to this.

I find the function provide_data is be defined but not using.

@xiaogehh I think the provide_data function was required and used by mxnet dataprovider. Whenever you create a new data iter, you need to define provide_data and provide_label function.

@jingpengw  Actually,it's not be very important weather it exists. I make a custom iter and find we only provide the next() function,then mxnet will return a Databatch object. Then we can feed this data to net

@xiaogehh It sounds like not consistent with the documentation. Maybe I am just out-dated. Anyway, good to know. Thanks.

@jingpengw  You are welcome~

@apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage.
For general "how-to" questions, our user forum (and Chinese version) is a good place to get help.

Since we now have NDArrayIter support HDF5, looks like the original issue has been resolved. Feel free to reopen if you still have problem loading HDF5 format data.


Correct Stern clusterrole

Note: this issue was imported from https://github.com/wercker/stern/issues/106, but it was originally created by Bulat-Gumerov...
Hey guys, I'm on EKS
I need to create clusterrole for developers which will allow tailing logs with stern and running commands inside k8s pods. I've created this clusterrole:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: dev-namespace
  name: exec-and-get-logs-from-pods
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

Developer can run kubectl logs, kubectl get pods and kubectl exec without any problem, but stern app fails with error: failed to set up watch: failed to set up watch: unknown (get pods). I don't want to give cluster-admin or admin role to our developers.
What's the correct clusterrole for stern?

Note: this comment was imported, but it was originally made by Bulat-Gumerov...
Any updates?

Note: this comment was imported, but it was originally made by mikhailadvani...
gives the same error even when watch is added to pods and pods/log


Taproot Leaf Script  - more than one entry

I have a question about the Taproot Leaf Script from BIP371:

Taproot Leaf Script
PSBT_IN_TAP_LEAF_SCRIPT = 0x15

The control block for this leaf as specified in BIP 341. The control block contains the merkle tree path to this leaf.

I believe the idea is that there may be several possible script paths that
the user is interested in collecting signatures for. The signer should
provide signatures for all that it can. The finalizer can throw an error if
multiple end up fully signed, or pick one at random to put in the tx, or
possibly other choices.
On Thu, May 5, 2022, 08:47 Jonathan Underwood @.***>
wrote:

Throw an error.
bip174 goal is to follow the spec, warts and all.
Psbt inside bitcoinjs-lib goal is to be a usable object with helpful
methods to deal with PSBT data. If we don't know how to deal with a valid
PSBT, throwing an error when we hit that wall is reasonable.
—
Reply to this email directly, view it on GitHub
https://github.com/bitcoinjs/bip174/issues/30#issuecomment-1118725101,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AVTUYXGITPKWEGTIBWEBYHTVIPUQRANCNFSM5VFM7OMA
.
You are receiving this because you were mentioned.Message ID:
@.***>

...warts and all

The spec does not say explicitly that PSBT_IN_TAP_LEAF_SCRIPT is a multi-value field.
However, since the <keydata> is not NONE I guess we have to assume multiple values. So the current implementation is correct.

...or possibly other choices.

hmm, I don't like unpredictability :)
probably I will start with allowing just one (for simplicity), and relax it later (backwards compatible)

Yeah, the change from "there's one script that must be satisfied to spend this coin" to "there are<PHONE_NUMBER>23 possible scripts and a keypath that you must satisfy one of" really breaks the paradigm for a lot of software.

@motorina0 I agree that seems inconsistent: it seems to allow to set all possible script leaves, but for signature PSBT_IN_TAP_SCRIPT_SIG indeed wants a single value, if I am reading the bip right 🤔
I agreed on throwing error if multiple script path are passed
cc/ @louisinger

@motorina0 I agree that seems inconsistent: it seems to allow to set all possible script leaves, but for signature PSBT_IN_TAP_SCRIPT_SIG indeed wants a single value, if I am reading the bip right thinking

I think the idea behind "multiple tapLeafScript" is mainly to be able to share several branches of your script tree via PSBT. I could add two leaves on my input, send you the psbt, and let u choose how the spend the coin.
PS: from my understanding PSBT_IN_TAP_SCRIPT_SIG needs three data:

the leafHash to identity which leaf the signature is associated to (and then get the controlBlock from tapLeafScript)
the public key associated with the sig
the signature

so theorically u could have several PSBT_IN_TAP_SCRIPT_SIG for the same leaf (in case of multisig for instance).

2-of-3 would be implemented as 3 branches of 2-of-2 with all combinations.
I would want to include both leaves where I am a participant.

FYI: I've updated the bitcoinjs-lib PR to use the taproot input fields (see comment)
Regarding the finalizer:

finalzieInput():

if tapKeySig is found then it will finalize using key-spend
alternatively it will search for the closest tapLeaf that has a signature. Closest means shortest path in the control block

finalizeTaprootInput()

this is a new public method. It allows specifying a tapLeafHash
if the tapLeafHash is provided, then it will try to finalize using that leaf


Fix interactive explore widget

Since the release of ipywidgets 6.0, injecting arbitrary javascript into an HTML widget is not allowed. This has broken lens.interactive_explore since it works by injecting the HTML output from an offline plotly plot generation into an HTML widget. We can consider fixing this by using plotly's GraphWidget, which adds complexity in the modifications of the plot, or completely bypassing plotly and rendering the plots with maptlotlib as proposed in #8.

Fixed by #17


[uniffi] Use a single welcome message in our API

Issues:
Addresses #81
Description of changes:
We have an expectation that all participants implement the functionalities the way we do (but not necessarily using mls-rs-uniffi as the underlying library). So if we make the assumption that single_welcome_message = true is always set, we can simplify the API.
Call-outs:
I'm not super sure what the implication is of throwing away the rest of the welcome messages if someone should send us more than one?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.52%. Comparing base (98cf9fe) to head (15230e4).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #129   +/-   ##
=======================================
  Coverage   89.52%   89.52%
=======================================
  Files         173      173
  Lines       31404    31405    +1
=======================================
+ Hits        28114    28115    +1
  Misses       3290     3290

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hey @mgeisler you always receive exactly one welcome message. Generating multiple is done if you want to split to make a welcome message for each new member to save on bandwidth on the download side. The server would figure out how to fan them out to individuals separately.

Hey @mgeisler you always receive exactly one welcome message. Generating multiple is done if you want to split to make a welcome message for each new member to save on bandwidth on the download side. The server would figure out how to fan them out to individuals separately.

In fact it's possible (I'd have to check with the RFC) that you can anyway take the single welcome message and split it up yourself into multiple welcome messages for different joiners, each smaller than the original one. I don't think there's any crypto binding all the pieces of the single welcome message together.

Thanks @tomleavy and @mulmarta for explaining!


Approve TXs prompts not showing up (received?) in Valora app

Hi all,
Francis from HaloFi here.
We have many users reporting connectivity issues w/ Valora and I Valora is rolling out v. 1.64.2 which is expected to fix some of these issues.
I updated my Valora to the latest (1.64.2 on iOS 15.7.8, iPhone 7 plus), and assuming the new valora version fixes it, I'm still unable to approve transactions with Valora using rainbowkit-celo. I was able to reproduce this issue with both HaloFi and the Demo site (https://rainbowkit-with-celo.vercel.app/fee-currency).
I also contacted the Valora team via Help >> Contact menu option to share the logs of my valora session on my cellphone - I hope you can also get access to that w/ the Valora team.
the issue basically is that users are able to connect the wallet (accessing sites via mobile or desktop), but when a transaction that requires an approval is submitted, the approval request is never displayed in Valora.
I attached a small video displaying the steps I did in the Demo page to reproduce the issue. note that the site just keeps displaying Transaction sending in the left bottom corner, but nothing happens and the tx approval never arrived in my Valora wallet (cellphone not shown in video but was close to me at time of testing).
I'm also attaching an error that was displayed in the console after leaving the demo page open waiting for the Transaction sending process timedout. The screenshot attached contains the errors logged at the timeout moment.
Let me know if there's any other test/info we can perform/share to help out.
Thanks

there's nothing major in the video, other than the steps.
https://github.com/celo-org/rainbowkit-celo/assets/36540281/bec77989-9f00-47b8-9609-6ddb29eee1ff

fairly certain this will be cause by something further down in the stack as this package is just configs. but will double check that

sorry, we have a few conflicting priorities (and i believe valora looked into this) please let me know if there is still an issue or if you have found resolution


Flatten arch array to string

Architecture is an array so we should flatten it to a string for use with Sprintf.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Force merging to expedite release.


abseil-cpp compilation creates ABI compatibility issues between abseil-cpp and dependent packages

Describe the bug
The current abseil-cpp package is built with CMAKE_BUILD_TYPE set to None. This results in the package being built without NDEBUG being defined. This creates ABI incompatibilities between abseil-cpp and packages that depend on abseil-cpp that are compiled with NDEBUG defined.
I came across this issue when I was running tests for a custom grpc based service against Azure Linux. Each time I would run tests, the test executable would end up crashing with gdb tracebacks similar to this (sans traceback entries into proprietary code):
0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
1  0x00007ffff6729ed3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
2  0x00007ffff66ded86 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
3  0x00007ffff66c97e5 in __GI_abort () at abort.c:79
4  0x00007ffff63f2537 in absl::lts_20240116::raw_log_internal::(anonymous namespace)::RawLogVA(absl::lts_20240116::LogSeverity, char const*, int, char const*, __va_list_tag*) () at /lib/libabsl_raw_logging_internal.so.2401.0.0
5  0x00007ffff63f25ce in absl::lts_20240116::raw_log_internal::RawLog(absl::lts_20240116::LogSeverity, char const*, int, char const*, ...) ()
    at /lib/libabsl_raw_logging_internal.so.2401.0.0
6  0x00007ffff7b19280 in absl::lts_20240116::DeadlockCheck(absl::lts_20240116::Mutex*) () at /lib/libabsl_synchronization.so.2401.0.0
7  0x00007ffff7b1bb35 in absl::lts_20240116::Mutex::Lock() () at /lib/libabsl_synchronization.so.2401.0.0
8  0x00007ffff70113f0 in grpc_event_engine::experimental::BasicWorkQueue::Add(grpc_event_engine::experimental::EventEngine::Closure*) () at /lib/libgrpc.so.39
9  0x00007ffff700b19d in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::Run(grpc_event_engine::experimental::EventEngine::Closure*) () at /lib/libgrpc.so.39
10 0x00007ffff700b2f3 in grpc_event_engine::experimental::WorkStealingThreadPool::Run(absl::lts_20240116::AnyInvocable<void ()>) () at /lib/libgrpc.so.39
11 0x00007ffff70039e2 in grpc_event_engine::experimental::TimerManager::TimerManager(std::shared_ptr<grpc_event_engine::experimental::ThreadPool>) ()
    at /lib/libgrpc.so.39
12 0x00007ffff6ff27a0 in grpc_event_engine::experimental::PosixEventEngine::PosixEventEngine() () at /lib/libgrpc.so.39
13 0x00007ffff6fdd1c3 in grpc_event_engine::experimental::DefaultEventEngineFactory() () at /lib/libgrpc.so.39
14 0x00007ffff6fdc7a5 in grpc_event_engine::experimental::CreateEventEngineInner() () at /lib/libgrpc.so.39
15 0x00007ffff6fdc7d2 in grpc_event_engine::experimental::CreateEventEngine() () at /lib/libgrpc.so.39
16 0x00007ffff6fdc9c9 in grpc_event_engine::experimental::GetDefaultEventEngine(grpc_core::SourceLocation) () at /lib/libgrpc.so.39
17 0x00007ffff6fdcd9c in grpc_event_engine::experimental::(anonymous namespace)::EnsureEventEngineInChannelArgs(grpc_core::ChannelArgs) () at /lib/libgrpc.so.39
18 0x00007ffff6fdd06b in std::_Function_handler<grpc_core::ChannelArgs (grpc_core::ChannelArgs), grpc_core::ChannelArgs (*)(grpc_core::ChannelArgs)>::_M_invoke(std::_Any_data const&, grpc_core::ChannelArgs&&) () at /lib/libgrpc.so.39
19 0x00007ffff6f851e6 in grpc_core::ChannelArgsPreconditioning::PreconditionChannelArgs(grpc_channel_args const*) const () at /lib/libgrpc.so.39
20 0x00007ffff71655d0 in grpc_server_create () at /lib/libgrpc.so.39
21 0x00007ffff7a728c4 in grpc::Server::Server(grpc::ChannelArguments*, std::shared_ptr<std::vector<std::unique_ptr<grpc::ServerCompletionQueue, std::default_delete<grpc::ServerCompletionQueue> >, std::allocator<std::unique_ptr<grpc::ServerCompletionQueue, std::default_delete<grpc::ServerCompletionQueue> > > > >, int, int, int, std::vector<std::shared_ptr<grpc::internal::ExternalConnectionAcceptorImpl>, std::allocator<std::shared_ptr<grpc::internal::ExternalConnectionAcceptorImpl> > >, grpc_server_config_fetcher*, grpc_resource_quota*, std::vector<std::unique_ptr<grpc::experimental::ServerInterceptorFactoryInterface, std::default_delete<grpc::experimental::ServerInterceptorFactoryInterface> >, std::allocator<std::unique_ptr<grpc::experimental::ServerInterceptorFactoryInterface, std::default_delete<grpc::experimental::ServerInterceptorFactoryInterface> > > >, grpc::experimental::ServerMetricRecorder*) () at /lib/libgrpc++.so.1.62
22 0x00007ffff7a6d0ca in grpc::ServerBuilder::BuildAndStart() () at /lib/libgrpc++.so.1.62
...

This happened because the implementation of absl::Mutex is changed by the presence of an NDEBUG definition. When abseil-cpp is compiled without NDEBUG, this definition of absl::Mutex::Dtor() is defined in the shared library and meant to be called by the absl::Mutex destructor. This helps clear out deadlock info when NDEBUG isn't defined. However, when NDEBUG is defined by dependent packages, this definition of absl::Mutex::Dtor() is inlined in the dependent binary for use by the dependent package, resulting in the former definition being unused, and causing grpc and other dependent packages to abort sporadically (but often) at absl::Mutex entry points.
One of the prominent maintainers of abseil-cpp has mentioned this issue previously.
This pull request fixes the problem.
To Reproduce
Steps to reproduce the behavior:

Compile an executable that links against abseil-cpp, preferably something that creates and destroys absl::Mutex instances, and make sure to define NDEBUG when compiling.
Run the executable for some time, and wait for your executable to crash with a stack trace similar to the stack trace above.

(If the above doesn't illustrate the problem well enough, then I'll attempt to supply a minimal reproduction of the bug when I get a chance.)
Expected behavior
I expect applications that link to abseil-cpp to run without crashing.

I've attaching a reproduction of the crash to this issue.  You can compile the program using:
g++ -o repro repro.cpp -O0 -DNDEBUG $(pkg-config absl_synchronization --cflags --libs)

Running the compiled program using gdb yields the following:
danderson [ ~/projects/abseil-cpp-repro ]$ gdb ./repro
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./repro...
(No debugging symbols found in ./repro)
(gdb) run
Starting program: /home/danderson/projects/abseil-cpp-repro/repro
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/printers.py:1273: SyntaxWarning: invalid escape sequence '\d'
  self.typename = re.sub('^std::experimental::fundamentals_v\d::', 'std::experimental::', self.typename, 1)
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/printers.py:1302: SyntaxWarning: invalid escape sequence '\w'
  x = re.sub("std::string(?!\w)", s, m.group(1))
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/printers.py:1346: SyntaxWarning: invalid escape sequence '\d'
  self.typename = re.sub('^std::(experimental::|)(fundamentals_v\d::|)(.*)', r'std::\1\3<%s>' % valtype, typename, 1)
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:151: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?array<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:268: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?deque<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:312: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?forward_list<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:393: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?(__cxx11::)?list<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:508: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?vector<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:557: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?%s<.*>$' % self._name, class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:590: SyntaxWarning: invalid escape sequence '\d'
  if re.match('^std::(__\d+::)?__uniq_ptr_(data|impl)<.*>$', impl_type):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:592: SyntaxWarning: invalid escape sequence '\d'
  elif re.match('^std::(__\d+::)?tuple<.*>$', impl_type):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:654: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?unique_ptr<.*>$', class_type.tag):
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:723: SyntaxWarning: invalid escape sequence '\['
  m = re.match('.*\[(\d+)]$', str(self._elem_type))
/usr/lib/../share/gcc-13.2.0/python/libstdcxx/v6/xmethods.py:775: SyntaxWarning: invalid escape sequence '\d'
  if not re.match('^std::(__\d+::)?shared_ptr<.*>$', class_type.tag):
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
iteration 0
initializing mutex
locking mutex
destroying mutex
iteration 1
initializing mutex
locking mutex
[mutex.cc : 1418] RAW: Potential Mutex deadlock:

[mutex.cc : 1428] RAW: Acquiring absl::Mutex 0x7fffffffdb60 while holding  0x7fffffffdb60; a cycle in the historical lock ordering graph has been observed
[mutex.cc : 1432] RAW: Cycle:
[mutex.cc : 1446] RAW: mutex@0x7fffffffdb60 stack:
[mutex.cc : 1454] RAW: dying due to potential deadlock

Program received signal SIGABRT, Aborted.
0x00007ffff79960d4 in __pthread_kill_implementation () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007ffff79960d4 in __pthread_kill_implementation () from /usr/lib/libc.so.6
#1  0x00007ffff7945bde in raise () from /usr/lib/libc.so.6
#2  0x00007ffff792e832 in abort () from /usr/lib/libc.so.6
#3  0x00007ffff7f17527 in absl::lts_20240116::raw_log_internal::(anonymous namespace)::RawLogVA(absl::lts_20240116::LogSeverity, char const*, int, char const*, __va_list_tag*) () from /usr/lib/libabsl_raw_logging_internal.so.2401.0.0
#4  0x00007ffff7f175be in absl::lts_20240116::raw_log_internal::RawLog(absl::lts_20240116::LogSeverity, char const*, int, char const*, ...) ()
   from /usr/lib/libabsl_raw_logging_internal.so.2401.0.0
#5  0x00007ffff7fb21a0 in absl::lts_20240116::DeadlockCheck(absl::lts_20240116::Mutex*) () from /usr/lib/libabsl_synchronization.so.2401.0.0
#6  0x00007ffff7fb4ab5 in absl::lts_20240116::Mutex::Lock() () from /usr/lib/libabsl_synchronization.so.2401.0.0
#7  0x0000555555555255 in main ()

It looks like this was resolved by #10003; thanks for reporting this issue and contributing a fix!


TileMap tiledata corrupt when loaded from a PackedScene saved via ResourceSaver

,Godot version:
3.1 beta 3
OS/device including version:
Ubuntu 18.04.1 x86_64
Haswell-ULT Integrated Graphics Controller (driver: i915)
Issue description:
My project loads it's levels from a .JSON file and creates various Godot nodes via a GDScript. Once the level is loaded, it is saved into a .TSCN using a PackedScene, and the ResourceSaver class. This .TSCN is then used instead of the .JSON file for future loads. Unfortunately I have found that the TileMap's tile data gets garbled/corrupt/lost in the process of loading the .TSCN generated via the ResourceSaver.
Interestingly if you open the saved .TSCN file in the editor and make changes, these changes remain intact.  What I can gather from the source code is that the editor saves and loads scenes in the same way as my script, so I'm not sure what is causing the issue.
Steps to reproduce:
Programmatically create a TileMap.
Set various cells
Add the TileMap to a PackedScene
Save the Packed Scene via the ResourceSaver
Load the TileMap via the ResourceLoader
TileMap data no longer matches original.
Minimal reproduction project:
tilemap_save_load_bug.zip

The issue appears no longer reproducible in the current master branch, I always get "100% data matches".


No gradle tasks show up from me

Extension Name: vscode-gradle
Extension Version: latest
OS Version: Mac
VSCode version: latest stable
Describe the bug
No gradle tasks show up for me.  I can see them under tasks explorer but not gradle tasks
To Reproduce
Open java project.
Expected behavior
Gradle tasks to show up.
Screenshots

Output from "Gradle Tasks"
[info] Gradle server started
[info] Gradle client connecting to server...
[info] WebSocketClient: retry in 1000ms
[info] Connected to <IP_ADDRESS>:57312. Welcome client!
[info] CONFIGURE SUCCESSFUL in 9s
[info] CONFIGURE SUCCESSFUL in 128ms

That worked, thank you

Great, thanks for letting me know. This issue seems to be quite common, but I haven't been able to provide compatibility between the extensions. Perhaps if I have more time in the future I can figure out a nice solution.


API Key not Taking when Running Bot

Expected Behavior
Bot Should run and Spin Stops
Actual Behavior
Bot Spins a Stop then Crashes giving an API Error on Line 108:
[2016-08-15 18:09:45] [PokeStop] + 50 XP
[2016-08-15 18:09:45] [PokeStop] + 3 Poke Balls
[2016-08-15 18:09:45] [PokeStop] PokeStop is on cooldown for 4.94 minutes.
Traceback (most recent call last):
File "C:\Users\Root\Desktop\OpenPoGoBot-master\pokecli.py", line 380, in
main()
File "C:\Users\Root\Desktop\OpenPoGoBot-master\pokecli.py", line 373, in main
bot.run()
File "C:\Users\Root\Desktop\OpenPoGoBot-master\pokemongo_bot_init_.py", lin
e 118, in run
destination.target_alt
File "C:\Users\Root\Desktop\OpenPoGoBot-master\pokemongo_bot\stepper.py", line
74, in get_route_between
path_points = self.path_finder.path(from_lat, from_lng, to_lat, to_lng)
File "C:\Users\Root\Desktop\OpenPoGoBot-master\pokemongo_bot\navigation\path_f
inder\google_path_finder.py", line 16, in path
gmaps = googlemaps.Client(key=self.config.gmapkey)
File "C:\Python27\lib\site-packages\googlemaps\client.py", line 108, in __init
__
raise ValueError("Must provide API key or enterprise credentials "
ValueError: Must provide API key or enterprise credentials when creating client.
Steps to Reproduce
Start Bot Like Normal and Try to Run
Other Information
OS: Win 7

Git Commit:

Update to latest master as you are on an old version. You will need to update your config files accordingly.


Automated RFC workflow

Ideally, submitting a proposal should be as simple as writing the markdown document and clicking a button. The current workflow is a little too involved for non-techy users or even just people unfamiliar with the process here. It leaves too much room for potential mess such as:

Incorrectly submitted RFCs (e.g. to the wrong directory)
Inconsistent filenames and titles
Lack of or mis-linked discussions and fully rendered proposal links
People discussing the proposal in PR comments instead of in the discussion

Ultimately, I think we'd want to create a simple frontend for people submitting, editing and viewing proposal documents, but it would still use this repo behind the scenes. Devs would continue to use github for reviewing, discussing and managing RFCs.

i think that we should consider a few things

allowance of time for a person to correct mistakes and fully develop the PR
a. in my case i didnt full fledge out the PR and i was aware that there are going to be issues with the PR

people do need to be walked through github as most people do not use github

being reliant on github is a fairly big problem
and can easily create messes

I see using github as something to remove low effort submissions. If it's a simple change then a suggestions thread would be fine, for something more involved there should be more effort put into it.

@Teifion the same issue applies to people being aware of what is going on with the project
they have to now go on github RFC's to be aware of what is going on and how the conversations are proceeding

At the very least, I'd like to try and automate the discussion and rendered proposal links in the PR descriptions because I don't think anybody has got them right yet 😅


Disable deleting an alias or aliased index directly

Deleting an alias using es.indices.delete deletes the underlying
index instead. To avoid accidental deletion index status is checked
first:

If index name points to an alias, delete request fails with 400.
Aliases should be changed using the PUT endpoint.

If index HAS an alias, delete request fails with 400. This is a
safety measure to avoid accidental deletion of indexes that are
being used by the application.

Otherwise, the result of the index deletion request is returned.

:+1:


Segmentation fault when using twig.so (PHP 5.3.6 on Mac OS X Lion)

Mac OS X 10.7 (Lion) with PHP 5.3.6 (MAMP 2.0.2) with compiled twig.so produces "Segmentation fault" error when accessing non existing property like {{ app.user.company }} while user doesn't have such property.
Works:
{{ app.user.company|default('Company') }}
Segmentation fault instead of warning (like when not using as compiled module):
{{ app.user.company }}

It works for me. Can you give a simple script that does not work for you? At least, can you give us the Twig_Environment options you have?

Just checked. It is fixed and working well on latest twig version, but is a problem on twig 1.4, so bug is not relevant any more.


Update payment status

Upon sending a payment, UI should be updated with a completed status. This status update is triggered by a Ibex webhook and should be propagated downstream.

[x] Setup server that listens for changes from Ibex. Authenticate using a server secret
[x] Push message to Pubsub service + veriy message downstream
[ ] Create webhookUrl for local development

Currently testing with a manual http trigger. Look into options for local integration testing, such as ngrok

Currently getting client subscription errors (426 when calling ws-server at port 4000). Requires server investigation with little tribal knowledge.
Try testing via UI

UI is calling:
export const MyLnUpdatesDocument = gql`
    subscription myLnUpdates {
  myUpdates {
    errors {
      message
    }
    update {
      ... on LnUpdate {
        paymentHash
        status
      }
    }
  }
}

which is triggered on the backend by a different event type ("ACCOUNT_UPDATE") than what's used in the query in the previous merge. The Account Update event also requires more information about the transaction and receiver, which can be retrieved in a db query, though I haven't done a comprehensive check of all the required data and its sources. Fortunately, similar logic is done in flash/src/app/wallets/update-pending-invoices.ts which we can use as a reference.
This subscription also requires authentication, which means the ws-listener script needs to be updated in order to test.

For MyLnUpdates: https://github.com/lnflash/flash/pull/42


Fix HiveResult.toHiveString compatibility for Spark 4.0

:mag: Description
SPARK-47911 introduced breaking changes for HiveResult.toHiveString, here we use reflection to fix the compatibility.
Types of changes :bookmark:

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
build/mvn clean install -Pscala-2.13 -Pspark-master \
  -pl externals/kyuubi-spark-sql-engine -am \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.engine.spark.schema.RowSetSuite

before - compilation error
[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ kyuubi-spark-sql-engine_2.13 ---
...
[ERROR] [Error] /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala:30: not enough arguments for method toHiveString: (a: (Any, org.apache.spark.sql.types.DataType), nested: Boolean, formatters: org.apache.spark.sql.execution.HiveResult.TimeFormatters, binaryFormatter: org.apache.spark.sql.execution.HiveResult.BinaryFormatter): String.
Unspecified value parameter binaryFormatter.

after - UT pass
[INFO] --- scalatest-maven-plugin:2.2.0:test (test) @ kyuubi-spark-sql-engine_2.13 ---
[INFO] ScalaTest report directory: /home/kyuubi/apache-kyuubi/externals/kyuubi-spark-sql-engine/target/surefire-reports
Discovery starting.
Discovery completed in 1 second, 959 milliseconds.
Run starting. Expected test count is: 3
RowSetSuite:
- column based set
- row based set
- to row set
Run completed in 2 seconds, 712 milliseconds.
Total number of tests run: 3
Suites: completed 2, aborted 0
Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 58.55%. Comparing base (b810bcc) to head (6b3c743).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6404      +/-   ##
============================================
+ Coverage     58.51%   58.55%   +0.04%
  Complexity       24       24
============================================
  Files           653      653
  Lines         39975    40001      +26
  Branches       5489     5489
============================================
+ Hits          23391    23423      +32
+ Misses        14081    14077       -4
+ Partials       2503     2501       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Merged to master/1.9


Additional smoke tests for release 2.1

Many things have changed in preparation for release 2.1 and I think we need to do some ~integration~ smoke tests to ensure that the entire pipeline still works without nasty surprises.
Given our paper's focus on running across the tree of life, I suggest that sufficient integration tests would include:

one yeast dataset
one bacterial dataset
one other organism dataset

This could be combined with Align configurations with RiboViz release 2.1 configurations, example-datasets#76.

All pending 2.1 tasks need to be completed and merged into develop before this can begin. At the last developer meeting this was scheduled to be done by Wed 16 June. So that leaves Thu 17 June for people to do these tests and for any issues to be resolved for a release on Fr 18 June.

What I back in the past erroneously termed regression tests are in fact integration tests - see #290. If a run on a full size dataset raises an then this means that our automated tests are inadequate and need extended.

Apologies, I used the wrong terminology in referring to these as integration tests and have renamed the issue to "smoke tests". It is important to check on full-size datasets exactly to check for any unexpected bugs and, if necessary, write new automated tests.
Discussed with @FlicAnderson today and she will address.

I'll be running these on Eddie this afternoon or tomorrow morning after #367 and then #366 are merged into develop and I get the nod from @mikej888.
I'll try and set up some of the files required tonight to save time.
After these run (hopefully successfully), we're good to proceed to Release 2.1! :crossed_fingers:

Merged @acope3's PR #368 into develop after running tests as well.
Now currently running updated riboviz develop branch on Eddie on L-Sc_2014 (fungi) dataset (config in example-datasets branch update-configs-76).
I'm also currently setting up configuration files & job G-Hv_2020 (archaea) and W-Ec_2015 (bacteria) to get those jobs ready to run & submit.

I'm getting some issues on Eddie while trying to run more than one riboviz job submission at once - specifically trying to get G-Hv_2020 (archaea) and W-Ec_2015 (bacteria) running.  L-Sc_2014 is currently running and has no issues so far.
I had tested these G & W datasets via validate_only and such in an interactive session for each of these failing configs, so I'm fairly sure they should run now, and in the same interactive session I tested that they successfully got to the first process when I gave the 'full run' command, but I didn't necessarily wait about too long in an interactive session because there was lots to be getting on with.
This is the error from the job .e file after the job submission for the full dataset 'completed' (unsuccessfully) for G-Hv_2020, but it's the same story for W-Ec_2015:
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes
NOTE: Nextflow is trying to use the Java VM defined by the following environment variables:
 JAVA_CMD: /exports/csce/eddie/biology/groups/wallace_rna/anaconda/envs/riboviz/bin/java
 NXF_OPTS:

It doesn't really give more detail, but I'm wondering whether the issue is because I'm attempting to run two datasets at once, and these environmental variables is causing issues because they're already 'in use' in the system somehow?  @mikej888 is this a possibility?
I have a feeling that surely I must have previously had more than one dataset running at once on Eddie previously, but perhaps not since the environmental variables code came in. It is also possible that there's something about the job submission scripts or config files that's to blame, as well as my lack of familiarity running the latest config setups.
For now, I'm planning to leave the running job to complete (hopefully), then will try submitting another.  This does have implications in terms of knock-on time delays if I can only run them in serial, but I might try running one dataset locally (I successfully ran one sample from L-Sc_2014 earlier today, but it does take a while!) as another test.

@FlicAnderson sorry that's a mystery to me. I could understand it if you were running two Nextflow instances within a single interactive sessions but two batch jobs should surely be using entirely separate nodes.

Somehow the running job on Eddie seemed to have hung up at generateStatsFigs stage (I didn't see any file modifications for the last ~4 hours), so I tried running one of the other jobs instead. Same error.
Now I can't even get the first job that did start (L-Sc_2014) to run again, despite not changing anything in the job submission script or config files which ran as far as generateStatsFigs before hanging...   Think I'll have to give up with Eddie for tonight...  Not ideal :facepalm:
I'm not sure how to troubleshoot the error that's happening when I try and run the submitted jobs:
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes
NOTE: Nextflow is trying to use the Java VM defined by the following environment variables:
 JAVA_CMD: /exports/csce/eddie/biology/groups/wallace_rna/anaconda/envs/riboviz/bin/java
 NXF_OPTS:

In 'running things locally' news, it looks like there's deinitely a (?another?) bug...  :beetle:
For G-Hv_2020, there are 3 samples: FFC1, FFC2, and MNase.
I ran all samples in 1 go, running on my local laptop:
Completed at: 17-Jun-2021 20:01:45
Duration    : 2h 9m 5s
CPU hours   : 3.9
Succeeded   : 38

I get 3 folders in output, and the. pdfs actually look like the ones posted in @acope3 's comment on the dataset ticket.  So far so good.
I checked the .html outputs: They're all the same.
They are allocated 1 to each sample folder, and they're named correctly, but once you look at the .html, the Sample Name headings all show "MNase", regardless of which sample it's SUPPOSED to be.
The plots are all from the MNase sample as well, instead of being from the ones they should be.
The PDFs show that the correct outputs ARE being created, but it's not doing the right things in staticHTML (either the nextflow process, or the AnalysisOutputs side of things, I'm not sure).  I'm not sure if this is the weird "happens sometimes, I'm not sure why and can't replicate it" :ghost: :beetle: issue I had from #193 which seemed to have gone?
I didn't notice this issue earlier from the local run because I only ran one L-Sc_2014 sample previously (to test that it would run at all, and then I'd hoped I could move over to running on Eddie :cry: ).
The wrong-content .html output issue for G-Hv_2020 AND the weird "Error occurred during initialization of VM" error on Eddie for all 3 datasets have got me stumped for tonight (I have to stop for today now I think), and I think troubleshooting these issues could have rather unfortunate consequences for our 2.1 release schedule @mikej888 @ewallace @kavousan
I've started a local machine run of W-Ec_2015, and will check the .html outputs carefully when it completes. Depending on time, I might be able to start L-Sc_2014 (ALL samples this time!) locally before I go to bed.  (For what it's worth, the outputs for the 1-sample run of L-Sc_2014 I got earlier DO match outputs from a run of that dataset I did on the 21st of May or thereabouts, so that's a bit of a positive.)

Having thought a bit more about it and delved into the depths of the 'html output' issue ticket, I think this .html output issue IS the same intermittent bug I discovered previously in developing #193 and mentioned here.

heavy_minus_sign bug discovered an intermittent bug in my code for staticHTML process (at least I think that's where it's located). Frequently (but frustratingly not always?) the process seems to get the sample_id confused, so both my outputs WT3AT_output_report.html and WTnone_output_report.html contain data for the SAME sample_id.

Usually (?always?) the sample_id and data in both .html files is that of the second sample processed by staticHTML process. The files are put into their respective sample folders fine, but the content is wrong. Looking at the .command.sh content for each process, the sampleid placed in the rmarkdown::render() command is correct, but inside the markdown html output, print(params$sampleid) shows the wrong value.

I think that we talked about the possibility of this being a quirk of multi-core shenanigans in my Dell laptop, and I don't think Mike was able to reproduce it at the time.
So potentially, it might not be a blocker for anyone else.  It could just be the result of a momentary lapse of reason by my processor. I will try and run the same dataset again in the morning and see if the same thing occurs, or if it really is just an intermittent error.  If the latter, then we might still be ok to release?

For
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes

Some resources say this relates to ulimit which restricts the number of resources for a user. If I run this on my VM I get:
$ ulimit -v
unlimited

but I'd guess on EDDIE it might be constrained by whatever the job has asked for. Try adding ulimit -v to your job script and see what the value is.
In https://forum.image.sc/t/jvm-error-could-not-allocate-metaspace-solved/12543/5, a user of a Java package in a cluster has exactly the name error with the same number of bytes. A reply commented that:

the number of bytes refers to the max heap space of the JVM, which by default is 1G or<PHONE_NUMBER> bytes.

(For background, heap space in Java is used for dynamic memory allocation for Java objects and classes when a Java program runs. I've used Java software in the past that complained of running out of memory as too many objects were created and the memory in the heap was consumed. This was solved by setting a Java configuration parameter for it to request a larger heap. The size of the heap is ultimately restricted by the available memory of the host machine.)
The user then comments that:

Setting the virtual memory in the SGE batch file to this value:
#$ -l h_vmem=6g
fixed it for me.

and then commented that:

The reported maximum memory usage from the cluster was almost 2G. Apparently you have to at least double the amount of reserved virtual memory, therefore I chose 6G.

So maybe try a larger h_vmem value in the job script?
A FAQ, https://rce-docs.hmdc.harvard.edu/faq/why-do-i-get-java-error-could-not-reserve-enough-space-object-heap, also mentions "Could not allocate metaspace" in the context of a research computing environment. They comment that

To fix the error "Could not allocate metaspace", add the options "-XX:CompressedClassSpaceSize=m" to set the size of the class metadata memory allocation. In testing, 256m has been sufficient.

You should be able to set this via an environment variable in your job script
export JAVA_OPTS="-XX:CompressedClassSpaceSize=256m"
export _JAVA_OPTIONS="-XX:CompressedClassSpaceSize=256m"

and hopefully that might work.

For the staticHTML error I'll run some tests and see if I can replicate it locally. I'll also see if there is a way to find out where the mix up in samples might be arising from (maybe by looking at symbolic links from the work directory from staticHTML and where these link back to).

For the staticHTML error I edited vignette_config.yaml to have 4 samples:
  WTnone: SRR1042855_s1mi.fastq.gz
  WT3AT: SRR1042864_s1mi.fastq.gz
  NotHere: example_missing_file.fastq.gz # Test case for missing file
=>
  WT1: SRR1042855_s1mi.fastq.gz
  WT2: SRR1042864_s1mi.fastq.gz
  WT3: SRR1042855_s1mi.fastq.gz
  WT4: SRR1042864_s1mi.fastq.gz

I ran the workflow but it failed...
$ nextflow run -ansi-log false prep_riboviz.nf -params-file vignette/vignette_config.yaml
N E X T F L O W  ~  version 20.04.1
Launching `prep_riboviz.nf` [determined_yonath] - revision: f12ad0b907
samples_dir: .
organisms_dir: .
data_dir: .
[a3/d2702c] Submitted process > buildIndicesrRNA (yeast_rRNA)
[35/18a84e] Submitted process > cutAdapters (WT3)
[02/bc240a] Submitted process > buildIndicesORF (YAL_CDS_w_250)
[86/e8dcbb] Submitted process > cutAdapters (WT1)
[3e/96fd7c] Submitted process > cutAdapters (WT4)
[84/ab9772] Submitted process > cutAdapters (WT2)
[15/40958b] Submitted process > createVizParamsConfigFile
[5b/1542bf] Submitted process > hisat2rRNA (WT3)
[2d/c21c2e] Submitted process > hisat2rRNA (WT1)
[a9/d69f5c] Submitted process > hisat2rRNA (WT2)
[48/632c9b] Submitted process > hisat2rRNA (WT4)
[47/6d55e8] Submitted process > hisat2ORF (WT1)
[5c/e80ac6] Submitted process > hisat2ORF (WT3)
[81/2b33a1] Submitted process > trim5pMismatches (WT1)
[f2/c84fb4] Submitted process > trim5pMismatches (WT3)
[8c/3d5fe5] Submitted process > samViewSort (WT1)
[49/efde2f] Submitted process > samViewSort (WT3)
[22/c1ce66] Submitted process > outputBams (WT3)
[68/87455e] Submitted process > outputBams (WT1)
[4b/5d7543] Submitted process > makeBedgraphs (WT3)
[46/212c79] Submitted process > bamToH5 (WT3)
[db/b1b2f8] Submitted process > bamToH5 (WT1)
[d6/ddad35] Submitted process > makeBedgraphs (WT1)
[34/6f4651] Submitted process > hisat2ORF (WT2)
[ca/ade434] Submitted process > hisat2ORF (WT4)
[ea/5bedfc] Submitted process > trim5pMismatches (WT2)
[96/b500a7] Submitted process > trim5pMismatches (WT4)
[e6/0491ac] Submitted process > samViewSort (WT2)
[51/d0f3d8] Submitted process > samViewSort (WT4)
[48/701d7d] Submitted process > outputBams (WT4)
[38/318c2a] Submitted process > outputBams (WT2)
[de/6747be] Submitted process > makeBedgraphs (WT2)
[89/80600c] Submitted process > bamToH5 (WT4)
[29/197d9e] Submitted process > makeBedgraphs (WT4)
[4e/97dd36] Submitted process > bamToH5 (WT2)
[42/a88195] Submitted process > generateStatsFigs (WT1)
[b3/ceafa0] Submitted process > generateStatsFigs (WT3)
[8d/a9cead] Submitted process > generateStatsFigs (WT4)
[d8/18e68e] Submitted process > generateStatsFigs (WT2)
Finished processing sample: WT3
[d6/e26386] Submitted process > renameTpms (WT3)
[bd/6c08e6] Submitted process > staticHTML (WT3)
Finished processing sample: WT1
[2c/6f899c] Submitted process > renameTpms (WT1)
[5f/f3ab67] Submitted process > staticHTML (WT1)
Finished processing sample: WT4
[a0/302859] Submitted process > renameTpms (WT4)
[86/5273d9] Submitted process > staticHTML (WT4)
Finished processing sample: WT2
[00/c0b8dd] Submitted process > renameTpms (WT2)
[52/b5201e] Submitted process > staticHTML (WT2)
[65/76fe79] Submitted process > collateTpms (WT3, WT1, WT4, WT2)
Finished visualising sample: WT3
Finished visualising sample: WT1
[3f/764845] Submitted process > countReads
Finished visualising sample: WT4
Error executing process > 'staticHTML (WT2)'

Caused by:
  Process `staticHTML (WT2)` terminated with an error exit status (1)
...

Samples WT1, WT3 and WT4 succeeded but sample WT2 fails during staticHTML even though it is the same sample as sample WT4! So I compared the commands run for WT4 and WT2:
$ cat work/86/5273d9bbdf9177ef7028201d891aa6/.command.sh
#!/bin/bash -ue
Rscript -e "rmarkdown::render('/home/ubuntu/riboviz/rmarkdown/AnalysisOutputs.Rmd',params = list(verbose='FALSE', yamlfile='$PWD/config.yaml', sampleid='WT4', three_nucleotide_periodicity_data_file = '$PWD/3nt_periodicity.tsv', gene_position_length_counts_5start_file = '$PWD/gene_position_length_counts_5start.tsv', read_length_data_file='$PWD/read_lengths.tsv', pos_sp_rpf_norm_reads_data_file='$PWD/pos_sp_rpf_norm_reads.tsv' , gene_read_frames_filtered_data_file='$PWD/3ntframe_bygene_filtered.tsv', codon_ribodens_gathered_file='$PWD/codon_ribodens_gathered.tsv', sequence_features_file='$PWD/sequence_features.tsv' ), output_format = 'html_document', output_file = '$PWD/WT4_output_report.html')"

$ cat work/52/b5201e9d0b81d96cc7f1d4fdd9a85a/.command.sh
#!/bin/bash -ue
Rscript -e "rmarkdown::render('/home/ubuntu/riboviz/rmarkdown/AnalysisOutputs.Rmd',params = list(verbose='FALSE', yamlfile='$PWD/config.yaml', sampleid='WT2', three_nucleotide_periodicity_data_file = '$PWD/3nt_periodicity.tsv', gene_position_length_counts_5start_file = '$PWD/gene_position_length_counts_5start.tsv', read_length_data_file='$PWD/read_lengths.tsv', pos_sp_rpf_norm_reads_data_file='$PWD/pos_sp_rpf_norm_reads.tsv' , gene_read_frames_filtered_data_file='$PWD/3ntframe_bygene_filtered.tsv', codon_ribodens_gathered_file='$PWD/codon_ribodens_gathered.tsv', sequence_features_file='$PWD/sequence_features.tsv' ), output_format = 'html_document', output_file = '$PWD/WT2_output_report.html')"

and they are the same, except for the sample ID, as expected.

Execution halted

The failure for WT2 relates to
pandoc: AnalysisOutputs.utf8.md: openFile: does not exist (No such file or directory)

So the next step is to find out why that file isn't there.

From https://pkgs.rstudio.com/rmarkdown/reference/render.html, rmarkdown::render supports two parameters that we don't currently use but which are of interest:

intermediates_dir
Intermediate files directory. If a path is specified then intermediate files will be written to that path. If NULL, intermediate files are written to the same directory as the input file.
knit_root_dir
The working directory in which to knit the document; uses knitr's root.dir knit option. If NULL then the behavior will follow the knitr default, which is to use the parent directory of the document.

Given that AnalysisOutputs.md is in rscripts/ the comments on these parameters imply that any intermediate files will be created in rscripts/ also.
A web search provided a useful command, watch -n 0.1 "ls -ltr", which displays the results of ls -ltr every 0.1 seconds, so I monitored rmarkdown/ folder using:
$ watch -n 0.1 "ls -ltr" rmarkdown/

I reran the workflow. Two files prefixed by AnalysisOutputs repeatedly appeared and disappeared when rerunning the workflow. Also the workflow completed successfully!
So I'd more strongly suspect now that the reason for the mixed up sample IDs, and, for me, the failed run, is due to rmarkdown::render creating intermediate files with the sample name for different samples in the same directory and these are colliding.
Next step, try out those additional rmarkdown::render parameters.

I had a look-see on EDDIE as it was a Friday afternoon...
Error occurred during initialization of VM
Could not reserve enough space for 16777216KB object heap

16777216KB is 16GB.
4GB node (-l h_vmem=4G)
Request interactive node with 4GB memory for job:
$ qlogin -P bio_wallace_rna_riboviz -l h_vmem=4G

Free space on machine:
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           125G         39G         82G        1.6G        3.4G         83G
Swap:           15G        320M         15G

Virtual memory resource limit for user:
$ ulimit -v
4194304

4GB as requested in qlogin.
Run java -version using -Xmx to set maximum Java heap size:
$ java -Xmx1g -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
$ java -Xmx2g -version
...OK....
$ java -Xmx3g -version
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes # 1GB
$ java -Xmx4g -version
Error occurred during initialization of VM
Could not reserve enough space for 4194304KB object heap

I think the last is expected as it's asking for the entire free memory for the heap.
Try with -XX:CompressedClassSpaceSize=256m recommended in https://rce-docs.hmdc.harvard.edu/faq/why-do-i-get-java-error-could-not-reserve-enough-space-object-heap.
A comment on https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8043516 recommends that

Both XX:MaxHeapSize and XX:CompressedClassSpaceSize have to be specified.

A response to a StackOverflow question comments that the -Xmx option is equivalent to -XX:MaxHeapSize.
$ java -Xmx3g -XX:CompressedClassSpaceSize=256m -version
...OK...

The 3G heap is now accomodated.
$ java -Xmx4g -XX:CompressedClassSpaceSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for 4194304KB object heap

Set via environment variables:
$ export JAVA_OPTS="-Xmx3g"
$ export _JAVA_OPTIONS="-Xmx3g"
$ java -version
Picked up _JAVA_OPTIONS: -Xmx3g
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes
$ export JAVA_OPTS="-Xmx3g -XX:CompressedClassSpaceSize=256m"
$ export _JAVA_OPTIONS="-Xmx3g -XX:CompressedClassSpaceSize=256m"
$ java -version
Picked up _JAVA_OPTIONS: -Xmx3g -XX:CompressedClassSpaceSize=256m
...OK...

16GB node (-l h_vmem=16G)
Repeat steps from above but on an interactive node with 16GB memory for job:
$ qlogin -P bio_wallace_rna_riboviz -l h_vmem=16G

Free space on machine:
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           125G         39G         82G        1.6G        3.4G         83G
Swap:           15G        320M         15G

Virtual memory resource limit for user:
$ ulimit -v
16777216

16GB as requested in qlogin.
$ java -Xmx1g -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
$ java -Xmx2g -version
...OK...
$ java -Xmx3g -version
...OK...
$ java -Xmx4g -version
...OK...
$ java -Xmx8g -version
...OK...
$ java -Xmx12g -version
...OK...
$ java -Xmx14g -version
...OK...
$ java -Xmx15g -version
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes

Try -XX:CompressedClassSpaceSize=256m:
$ java -Xmx14g -XX:CompressedClassSpaceSize=256m -version
...OK...
$ java -Xmx15g -XX:CompressedClassSpaceSize=256m -version
...OK...

The 15G heap is now accomodated.
$ java -Xmx16g -XX:CompressedClassSpaceSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for 16777216KB object heap

4GB node (-pe interactivemem 4 -l h_vmem=4G
Use qlogin command from run-on-eddie.md:
$ qlogin -pe interactivemem 4 -l h_vmem=4G -P bio_wallace_rna_riboviz
$ free -h
              total        used        free      shared  buff/cache   available
Mem:            62G        3.0G         58G        195M        1.3G         58G
Swap:           15G        1.1G         14G
$ ulimit -v
16777216

16GB despite 4GB being requested in qlogin. It's because I requested 4 cores.
$ java -Xmx1g -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
$ java -Xmx2g -version
...OK...
$ java -Xmx3g -version
...OK...
$ java -Xmx4g -version
...OK...
$ java -Xmx8g -version
...OK...
$ java -Xmx12g -version
...OK...
$ java -Xmx14g -version
Error occurred during initialization of VM
Could not allocate metaspace:<PHONE_NUMBER> bytes
$ java -Xmx15g -version
Error occurred during initialization of VM
Unable to allocate 491520KB bitmaps for parallel garbage collection for the requested 15728640KB heap.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
$ java -Xmx16g -version
Error occurred during initialization of VM
Could not reserve enough space for 16777216KB object heap

Try -XX:CompressedClassSpaceSize=256m:
$ java -Xmx14g -XX:CompressedClassSpaceSize=256m -version
...OK...
$ java -Xmx15g -XX:CompressedClassSpaceSize=256m -version
Error occurred during initialization of VM
Unable to allocate 491520KB bitmaps for parallel garbage collection for the requested 15728640KB heap.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
$ java -Xmx16g -XX:CompressedClassSpaceSize=256m -version
Error occurred during initialization of VM
Could not reserve enough space for 16777216KB object heap

Ideas for future exploration
Specify -Xmx and XX:CompressedClassSpaceSize in JAVA_OPTS and _JAVA_OPTIONS as exported environment variables in the job script.
Use -XX:CompressedClassSpaceSize=256m
Use -Xmx (maximum heap size) initially as half the h_vmem value.
See what happens!

Raised Investigate Java and Nextflow on EDDIE Could not allocate metaspace issue #375 (and moved my comments there)

@FlicAnderson I've merged the #372 fix into develop and main. It'd be useful if you could complete the tests you'd planned. Once done (and hopefully with no more issues) I think we can then do the 2.1 release.

As discussed at Dev meeting on 30th June, I'm planning to try running again on Eddie for the three datasets I'd aimed for previously, and will have time for this next week (beginning 5th July). This will then allow Mike to build release 2.1

I have been running multiple datasets on eddie when checking upgraded config files. I did not encounter any unusual errors, and any that I did were down to pathing errors. All runs completed successfully once paths had been corrected.
The majority of this was run in the develop branch with commit:
commit 106a204bc64525db8de7d18e99d5421be1246394
Merge: 68bbe29 bf06a94
Author: Michael Jackson<EMAIL_ADDRESS>Date:   Mon Jun 21 09:10:32 2021 +0100

I also ran a series of datasets with updated config files on Eddie. The only error encountered was related to incorrect pathways. All runs were successful and produced the expected outputs.
The datasets were run from the develop branch with the same commit:
commit 106a204bc64525db8de7d18e99d5421be1246394
Merge: 68bbe29 bf06a94
Author: Michael Jackson<EMAIL_ADDRESS>Date:   Mon Jun 21 09:10:32 2021 +0100

I think this is done and can be closed. Many of us have used R2.1, extensively.


docs: Improve merge documentation

Add example indicating behaviour with values set to 'undefined'

@davidchambers I've seen "other libraries set a bad precedent".  In light of your's and @CrossEye's  comments above, I'm closing this PR.

I appreciate your effort, @rpwjanzen. Please continue to draw attention to parts of the documentation which could be improved. :)

And to anything else you notice!
Thank you for your attention to detail.


sap.m.DatePicker year picker not returning the same year selected

OpenUI5 version: latest (1.87)
Browser/version (+device/version): Any
Any other tested browsers/devices(OK/FAIL):
URL (minimal example if possible):
https://openui5.hana.ondemand.com/entity/sap.m.DatePicker/sample/sap.m.sample.DatePicker
User/password (if required and possible - do not post any confidential information here):
Steps to reproduce the problem:

Open Date Picker

Click on the year to change year

Select a year

What is the expected result?
Expected March 2027
What happens instead?
I get March 2028

Any other information? (attach screenshot if possible)

Hello @jbulda ,
Unfortunately, i cannot reproduce the issue. Which DatePicker have you opened or what properties are configured?
Please, also check if there are any other prerequisites.
Thank in advance,
Tereza

Hi @terezamch ,
Thank you for looking at this issue. I literally just used the first picker in the Samples (https://openui5.hana.ondemand.com/entity/sap.m.DatePicker/sample/sap.m.sample.DatePicker)
I tried changing the year to 2027 and it returned 2028. Also, I noticed that when selecting a year, the initial selected value is 2020 (see screenshot from step 3). There are no other prerequisites.

Hello @jbulda ,
I tried again, but with no success, I asked my colleagues to reproduce it, but again unfortunately with no success.
Could you give more info about the OS, browser and browser version you are using?
I have made a video showing what i do, please correct me if something is wrong.
https://user-images.githubusercontent.com/12902122/110098822-47a6f080-7da9-11eb-8b07-388f71e9b89b.mp4
Regards,
Tereza

Hi @terezamch,
Thank you, again, for taking the time to look at this issue.
I'm using a macOS v10.14
Chrome v89.0.4389.72
Here's how it looks like:
https://user-images.githubusercontent.com/21362575/110129226-b7ab7b80-7d95-11eb-8f99-b9c9af34aff1.mov

Hello @jbulda ,
I updated my Chrome to use the mentioned version but sill cannot reproduce the issue.
I have already tried using Safari and Firefox, but with no luck.
The only difference in my setup compared to yours is the macOS - i am using 11.2.2
Is it possible for you to update it? Unfortunately I cannot downgrade...
Thanks and best regards,
Tereza

Hi @terezamch unfortunately, I cannot upgrade...

Hi @jbulda ,
I was not able to reproduce it either. I'm using the same Chrome version.
In order to support you, we'd need a stable reproduce path.
Please try with another machine to check if it is reproducible there.
Cheers

Hello @jbulda
Are you still able to reproduce this issue with the DatePicker? Here's the sample in version 1.87, and here's the sample in the latest release.
Kind regards,
Georgi

Closing this issue due to inactivity. Feel free to reopen if you have an update on the bug!
Regards,
Georgi


Check policy directly in submit

There is no need to stuff the policy in the context. In the places that we need it, we can either make another API call, or, as the case with Submit is, we already have what we need in the certProject.

@bcrochet if we're going to walk back stuffing the policy into the context, should we also revert any other changes made in the same PR? E.g. the context helpers in the policy package?

@bcrochet if we're going to walk back stuffing the policy into the context, should we also revert any other changes made in the same PR? E.g. the context helpers in the policy package?

I'm ambivalent. You say remove, I'll remove.

@bcrochet if we're going to walk back stuffing the policy into the context, should we also revert any other changes made in the same PR? E.g. the context helpers in the policy package?

Removed.

/retest


client authentication

any chance to implement client authentication?

I should be able to add a callback for that pretty easily, yeah.

any updates on this?

Does 845d60ed5d2721d50017a6131fa7d356c838d610 help?

Hmm, I think that the Rust way would be to provide a Trait.
See for example: https://github.com/bkchr/picoquic-rs/blob/c38d6a8a683e4b697fe0d2e71e14f926d733710c/src/verify_certificate.rs#L10
Also you don't expose any way to match the client certificate to a connection?

Hmm, I think that the Rust way would be to provide a Trait.

That doesn't seem rustier to me. A trait defined with nothing but a single method like that is functionally equivalent to a closure, but requires more boilerplate.

Also you don't expose any way to match the client certificate to a connection?

Interesting point. What's the use case? The obvious case of checking for a certain private CA, for example, doesn't to require that afaict.

In Rust you even have cases where a Trait does not have any function at all. Maybe it is just a matter of taste.
For example when you want to use the certificate of a connection for other use cases, it is important to match a Connection to its certificate. You could for example use the certificate as an identifier of the connection.

I feel like that would be better solved by exposing a certificate accessor on connections, similar to the ones currently exposed for SNI. The authentication callback should be concerned solely with the question of whether the certificate is acceptable at all.

In lieu of any further comment from @aep, I think this is adequately solved for now.


Rename the MC class

New name: CFLARE - Clustering and Filtering of Left and Right Eigenvectors.

Will be implemented via #103

Implemented via #103


Functions update

This PR:

updates the color() function to match the tachyons styleguide
updates the color-variants() function so you can now pass it more color functional classes as childrend. And you don't have to specify the color each time
adds the color-states() function

Hey my friend, I am finally here :D
I would merge this, but the tests are broken. Do you want me to have a look at it and fix?

@caesarsol W H A T
Anyway, should be fixed now

I had lost every hope

AHAHAH


Define Config Schema for Parsing / Transforming layer

{[
  "id": 0x32,
  "scale": 100,
  "transformer": "little_endian",
  "start": 7,
  "length": 16,
  "key": "inverter_temp"
],
[
  "id": 0x03,
  "scale": 10,
  "transformer": "little_endian",
  "start": 7,
  "length": 8,
  "key": "motor_temp"
],
[
  "id": 0x04,
  "scale": 1,
  "transformer": "vw_module",
  "start": 7,
  "length": 8,
  "key": "module1_temp"
]
}

Would we then register a 'transformer' function to produce the appropriate UI output JSON?

I was thinking the transformer would just return a value, be it a string, number, array etc and we'd have something else that used the value to generate the message.

OK, yeah. Makes sense. So we have a single place where the messages are produced, based on a stream of incoming data from the transformers?

Pretty much, yea.


Run README files through Grammarly

This PR tries to fix a few typos I found when reading the README files. I reported these on this issue: #1352
I tried to be very conservative here so I haven't fixed all the issues, but the ones I did I think are relevant.
For example, I ignored many suggestions to add commas or hyphens because these don't affect the readability.

@toast-gear thanks for the help, just applied your suggestion.


Fixed checking skip_downloads condition.

skip_downloads variable doesn't work for ansible 2.6.1.
Please fix it.

ci check this

Error from CI is:
"/opt/bin/pypy/bin/pypy: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

Is it really my PR problem?

rebase code . fix in #3034


there si no way to add extra configurations to oauth server (custom grant_types)

I need to add a custom grant type to the instances of oauth2server used in zf-mvc-auth.
There is no way to do this because there is no shared registry for all the oauth2 servers.
The class ZF\MvcAuth\Factory\DefaultAuthenticationListenerFactory creates a new oauth server every time that is called.
Also,  ZF\OAuth2\Factory\OAuth2ServerFactory creates a new instance every time that is called.
This could be fixed if zf-mvc-auth provides a way to add custom grants by configuration.

Not sure what do you mean by 'custom grant_types'. Do you mean really custom grant type implemented from scratch or just custom storage for one of the already existing? For the latter, you can simply set 'grand_types' property in your oauth config in autoload folder, and set grand types which you needs:
'grant_types' => array(
 'client_credentials' => true,
 'authorization_code' => true,
 'password'           => true,
 'refresh_token'      => true,
 'jwt'                => true,
 )

I mean custom grant types implemented from scratch.
I am doing Google and Facebook login integration, I have a custom grant
type to manage the google hybrid type of login.
Once that the web client get a code from Google its sended to the oauth
server using a custom grant type key. Then the server validate this code
and get an access token from google.
I was looking the code to figure out how to achieve this. And can't find a
clean way. the grant_types configuration key maybe should accept custom
grant types factories or instances, what do you think?
El 29/04/2015 04:10, "TarasHots"<EMAIL_ADDRESS>escribió:

Not sure what do you mean by 'custom grant_types'. Do you mean really
custom grant type implemented from scratch or just custom storage for one
of the already existing? For the latter, you can simply set 'grand_types'
property in your oauth config in autoload folder, and set grand types which
you needs:
'grant_types' => array(
'client_credentials' => true,
'authorization_code' => true,
'password' => true,
'refresh_token' => true,
'jwt' => true,
)
—
Reply to this email directly or view it on GitHub
https://github.com/zfcampus/zf-mvc-auth/issues/70#issuecomment-97329673.

@cape I'm working on a solution now; it will encompass changes on both zf-oauth2 and zf-mvc-auth. In each case, the factory returned by the ZF\OAuth2\Service\OAuth2Server service will cache OAuth2\Server instances, making it safe for you to do this:
$factory = $services->get('ZF\OAuth2\Service\OAuth2Server');
$server = $factory(); // application-level zf-oauth2 server, OR
$server = $factory('named-oauth2'); // per-api server instance, as provided in zf-mvc-auth

$server->{call a method}(...);

That should likely resolve your issue. I'll update this issue with pull requests for you to track shortly.

@cape #71 and zfcampus/zf-oauth2#95 will address this. I've also described in #92 how you can create a delegator factory for automating injection of your custom grant types once those patches are merged (will be within the hour).

Thanks!! This will help me a lot.
I will test this asap!!!
Thanks again
El 29/04/2015 19:16, "weierophinney"<EMAIL_ADDRESS>escribió:

Closed #70 https://github.com/zfcampus/zf-mvc-auth/issues/70 via 814c630
https://github.com/zfcampus/zf-mvc-auth/commit/814c630b5835968bf484b8d6745bee241b19d5b1
.
—
Reply to this email directly or view it on GitHub
https://github.com/zfcampus/zf-mvc-auth/issues/70#event-293510911.


Cypress e2e testing

closes #227 depends on #279
prerequisites:

[x] branch is up-to-date with the branch to be merged with, i.e. develop
[x] build is successful
[x] code is cleaned up and formatted
[ ] tested with Firefox 52, Firefox 57+, Chrome 64+, MS Edge 16+

Summary
wip of using cypress to run e2e tests.
Note: as part of this the location of the typings have changed to /build/src

local testing
npm run compile
npm run build:prod
npm run cy:open

I added a simple test for #275 we can add more tests on the fly

@sgratzl How should we proceed with this PR? Should I wait for further implementation or merge it once you addressed my two minor change requests?

merge then we can start using it


thread 'main' panicked at 'not currently active!?', src/tools/cargo/src/cargo/core/resolver/mod.rs:832:20

Problem
When building the linked minimal reproducible example cargo panics.
This bug got reported before in #11012 . However, they were not able to reproduce the error.
I found a minimal reproducible example.
Steps

git clone https://gitlab.com/w0lff/cargo-issue-11012.git
cargo build

Possible Solution(s)
No response
Notes
The = in the version field of the wayland-client crate is needed now because the maintainer published newer versions. Without it cargo fails with an error and does not panic anymore.
Version
cargo 1.65.0 (4bc8f24d3 2022-10-20)
release: 1.65.0
commit-hash: 4bc8f24d3e899462e43621aab981f6383a370365
commit-date: 2022-10-20
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Arch Linux Rolling Release [64-bit]

Thanks for your report!
It is definitely a bug that Cargo panics. From the other angle, it also does things partially correctly. Your package depends on two SemVer-compatible versions of a single package, which is not currently possible in Cargo.
I've got a minimal example in Cargo's own test dialect more or less the same as your example. Let's investigate it more!

Expand to see the example
#[cargo_test]
fn mismatched_version2() {
    Package::new("qux", "0.1.0-beta.1").publish();
    Package::new("qux", "0.1.0-beta.2").publish();
    Package::new("bar", "0.1.0").dep("qux", "=0.1.0-beta.1").publish();
    let p = project()
        .file(
            "Cargo.toml",
            r#"
                 [package]
                 name = "foo"
                 version = "0.1.0"

                 [dependencies]
                 bar = "0.1.0"
                 qux = "=0.1.0-beta.2" # still panic without `=`

                 [patch.crates-io]
                 qux = { path = "qux" }
            "#,
        )
        .file("src/lib.rs", "")
        .file(
            "qux/Cargo.toml",
            r#"
                [package]
                name = "qux"
                version = "0.1.0-beta.1"
            "#,
        )
        .file("qux/src/lib.rs", "")
        .build();

    p.cargo("check")
        .with_status(101)
        .with_stderr("")
        .run();
}

I am seeing the same issue as well with a combination of crates, but it is hard to debug given that cargo panics and does not update lock file.
Any updates on fixing the panic?

Fixed by #11770 in 1.70.0. Thanks for the report and reproducer!


Automate release cut, release-notes

If this commit applied no manual actions will be needed on release cut process apart from tag push
Issue: MM-36354

Summary
Automate release cut, release-notes
Ticket Link
https://mattermost.atlassian.net/browse/MM-36354
Release Note

Automates release cut.

The files .circleci/config.yml, Makefile, scripts/go_install.sh, scripts/go_install.sh, scripts/notify.sh, scripts/notify.sh, scripts/release.sh, scripts/release.sh are in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers

The files .circleci/config.yml, Makefile, scripts/notify.sh, scripts/notify.sh, scripts/release.sh, scripts/release.sh are in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers

The files .circleci/config.yml, Makefile, scripts/notify.sh, scripts/notify.sh, scripts/release.sh, scripts/release.sh are in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers


Update README.md

Fix typo.

Coverage remained the same at 97.895% when pulling cea7f77c6507d172ad889707dd5502b817c262b1 on vinceve:patch-1 into 7cc5e229f0794a2de5acf2442ded858eb6cbd161 on ruby-grape:master.

🐱 🐈 😹 😸 😼

thanks @vinceve … 😹


Introduce in-app update ( at least notification like Chrome )

This makes a lot of sense. Google can F themselves. Many such not-supposed-to-be-violated apps have in-app update like xda-labs which introduce in-app update.
Please make this soon if possible, especially at times like this.

Possible Play Store violation regarding binary distribution post-install: https://support.google.com/googleplay/android-developer/answer/9934569?hl=en

For Play Store builds, we could use the in-app updates feature.


build: GOEXPERIMENT=unified fails using Go 1.17 as bootstrap toolchain

% GOROOT_BOOTSTRAP=$HOME/sdk/go1.17 GOEXPERIMENT=unified ./make.bash
go: unknown GOEXPERIMENT unified
Building Go cmd/dist using /Users/rsc/sdk/go1.17. ()
go: unknown GOEXPERIMENT unified
%

I'm not sure what's going on here. It looks like maybe GOEXPERIMENT should be cleared during the use of the bootstrap toolchain in make.bash?

AFK now, but IIRC, there's an issue tracking this issue already (and also Matthew's CL to address it).

Thanks. I don't see it here. https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+goexperiment+unified

Looks like maybe CL 344512 is the fix.

This doesn't fail now. Probably fixed by CL https://go-review.googlesource.com/c/go/+/395879