Document Data Flow from Scrape to Sale

As a team member, I need to create a comprehensive document that outlines the flow of data from the initial email scrape of a receipt to the final sale. This document will serve as a reference for stakeholders, detailing each step in the process and the systems involved.
Acceptance Criteria

Identify and document each stage in the data flow process from email scrape to sale.
Describe the role and functionality of each component or system involved.
Include details on data transformations, processing, and any intermediary steps.
Specify data formats and communication protocols between systems.
Incorporate relevant privacy and security considerations at each stage.

1. Email Upload:

User Action: Upload the email body and its attachments through the TIKI platform.
Behind the Scenes: The email is securely transmitted to our system encrypted through HTTPS, preventing its interception by 3rd parties.

2. Publisher Identity Verification and Data Integrity Check:

What Happens:

We ensure the identity of the publisher who submitted the email.
We check that the email hasn't been tampered with during its journey to our system.

Why:

To confirm that the email comes from the publisher.

Outcome:

If the verification fails, the request is rejected.
If everything checks out, we move to the next step.

3. User's License Check:

What Happens:

We confirm whether the user has a valid license to submit its receipt data.

Why:

To ensure compliance with legal agreements and data usage policies.

Outcome:

If the user has a valid license, we proceed; otherwise, the request is rejected.

4. Create a Request Record:

What Happens:

We generate a unique identifier for this specific email submission.

Why:

To keep track of this specific transaction for future reference.

Outcome:

An ID for this request is created with the status "in progress". No other data is saved from the request.

5. Text Extraction from Email:

What Happens:

We use Amazon Textract to convert the text from the attahcments into plain text data.

Why:

To extract meaningful information from the receipt email.

Outcome:

The email attachments are discarded after submission
The text content is obtained from the attachments.

6. Convert Text to Structured Receipt Data:

What Happens:

We structure the extracted attachments text and the email body into a standardized format suitable for our data repository (TIKI Ocean).

Why:

To organize the information in a way that our system can efficiently process and store.

Outcome:

The text is transformed into structured Receipt Data.
After transformation, the text and the email body are discarded.

7. Publish Receipt Data to TIKI Ocean:

What Happens:

The transformed Receipt Data is sent to TIKI Ocean, our central data repository.
The request record created earlier is updated with the status as "stored".

Why:

To store the receipt data securely and make it accessible for data buyers.

Outcome:

The receipt data is successfully stored in TIKI Ocean, and the transaction is recorded for future reference.


Glaciers: Modernize example

[x] Modernized the example
[x] Updated the environment files

Ok @Azaya89 I think I prefer the approach where we instantiate the template at the very end only. The other way, with pn.extension(template='...') and .servable(area='main/sidebar') is useful but confuses the story a little too much I find. So please remove these bits of code, to focus on building the template at the very end of the notebook.

Ok @Azaya89 I think I prefer the approach where we instantiate the template at the very end only. The other way, with pn.extension(template='...') and .servable(area='main/sidebar') is useful but confuses the story a little too much I find. So please remove these bits of code, to focus on building the template at the very end of the notebook.

What part of the notebook are you referring to pls?

Hmm the layout looks a bit weird on the website.

Therefore, for examples that deploy an app, could you please share a screenshot/GIF of the app in the PR?

Here's how the app looks like now

I built the site locally and it looks much better, thanks for your last changes. I'll update the branch and merge if the dev site site looks similar.


Added new (yet unproven) blocking Read trait for a rngs

Implements #38 . Identical custom trait already implemented in nrf51-hal crate and used in
examples (directly and to seed a PRNG) in the microbit crate.
Signed-off-by: Daniel Egger<EMAIL_ADDRESS>

I'm not immediately clear on how this is to be used, is the intention with this to block until entropy is available or return an error in the case it's not?
And given entropy is often limited on the platforms we use, is there a reason this is better served by a blocking call than by polling on a non-blocking one (which is then easily wrapped to block)?

I'm not immediately clear on how this is to be used, is the intention with this to block until entropy is available or return an error in the case it's not?

Almost. The Error is meant for situations where no entropy can be generated at all, e.g. if there's not random generator or it is not available for some reason (like being used by another hardware block).

And given entropy is often limited on the platforms we use, is there a reason this is better served by a blocking call than by polling on a non-blocking one (which is then easily wrapped to block)?

Well, if you need entropy then you often need a specific amount of it and I find it a bit inconvenient to handle such a situation with async calls other than blocking on the spot. Different platform will also yield a different numbers of bytes per round, e.g. with NRF51 you'll get 1 byte per round while the STM32s will yield 4 byte per round.
If you need a larger amount of data I'd recommend setting up an interrupt handler and simply using the blocking call with the amount of data a platform will yield. Of course we still don't have any support whatsoever to properly include interrupt setup and handling in the HAL drivers so that requires some dancing around...
One could certainly craft an async interface, too, but at the moment I don't see too much benefit in that.

The benefit I can see to an async interface is interaction with nonblocking application models, such as those based on await! or futures.
The nb crate is designed for exactly this kind of interoperability, and making the trait itself compatible should be as simple as replacing the Result with an nb::Result

@austinglaser And where would you store the data in the meantime? The API is modelled like the other blocking APIs in that you pass in a buffer to be filled. Otherwise you can only return a single value in the result.

@therealprof That's a good point -- I hadn't thought through that aspect enough.
What about a trait (eventually, and in addition to this one) which returns a fixed-size random value?
That fixed-size value could be (taking a page out of the spi traits' book) determined on a per-implementation basis, or it could simply be a u8 -- meaning that on hardware like the STM32, you might just hold on to three bytes of data to give out on later calls.
I think I favor the simple u8 approach, because it makes filling a buffer with values simple at the cost of a little more complexity for the implementer.
Something like this?
// rng.rs
pub trait Read {
    type Error;

    fn read(&mut self) -> nb::Result(u8, Self::Error);
}

But this is getting beyond the scope of discussion for this particular pull request, I think.

@austinglaser A single u8 might be somewhat limiting if that's not what you need in a go. I'm not sure how you would store the data internally in an implementation; once you leave the function any previous values stored on the stack would be gone, unless you have a (yuck!) static buffer.

@therealprof I'd say that forcing a user to block is limiting in a different way -- at the trait level, at least, it makes sense to allow both usage patterns.

I'm not sure how you would store the data internally in an implementation

You'd store the data in a context struct you pass back into the function each time. If that doesn't sound terribly ergonomic, you could use futures or generators, which do much the same thing behind the scenes but (especially in the generator case) are a little cleaner to use.

For a semi real-world example, take a look at the docs for the nb crate. Note that the LED "thread" in each case has some mutable data (state) which is maintained across resumptions/polls of the asynchronous function.

I'd say that forcing a user to block is limiting in a different way -- at the trait level, at least, it makes sense to allow both usage patterns.

Blocking on a MCU is relative anyway since you can always interrupt the wait. But sure, if you have a use case for partial random data being available whenever it shows up that an async interface can easily be added.

You'd store the data in a context struct you pass back into the function each time. If that doesn't sound terribly ergonomic, you could use futures or generators, which do much the same thing behind the scenes but (especially in the generator case) are a little cleaner to use.

You're absolutely right that this is not ergonomic. ;) I'm familiar with the concepts of futures and generators in general. I'm still not sure about the use case in with a TRNG; maybe if you throw in DMA to automatically build up a pool from which you can deplete. But with a register based implementation there's very little point in having this; instead if you need large amount of data, use the TRNG to seed a PRNG and use that to generate random numbers whenever you need them.
But hey, feel free to propose the non-blocking version of this trait. I'd even provide a non-blocking implementation, just for the fun of it. 😉

But hey, feel free to propose the non-blocking version of this trait. I'd even provide a non-blocking implementation, just for the fun of it.

Fair enough!

Oh and kind of off-topic but there are a few traits that look very similar to each other and that look similar to some of the API in std::io::{Read,Write} so I've been wondering if it would make sense to have some core blocking::ReadExact trait with method fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Self::Error> and then either (a) have traits like blocking::Rng be a super trait over that or (b) have blocking::Rng become a marker trait then the user would have to use where RNG: blocking::ReadExact + Rng.

@japaric Would you like me to change the name to read_exact() to be in line with the std::io::Read implementation?


stat: implement Poisson rejection sampling.

I'm guessing I may have missed some feedback as it was originally intended. Please let me know what I let fall through this last commit and I will be sure it get it in the next one.


Where is camera_webcam.cc? How to change the cameras id?

Describe the bug
I want to test it on my laptop,but the camera of my laptop can not be recognized.So I want use two webcams,but only one of them which is used to monitor the driver's face can be recognized,I want to change the ids of the camera in the program,but can not find camera_webcam.cc which is used to change the camera's id in the past versions.What should I do?
OS Version
ubuntu 20.04
openpilot version or commit
No response
Additional info
No response

See #24590


Add type magic!

Using phantom types we can keep track of whether a set of flags is for output or if it is valid for both input and output.
It's been a while since I've done this sort of thing so I'd like to revisit this. I think of_int should be named unsafe_of_int and the phantom type names could likely be clearer.

I agree with the intention of making it harder to mess with the flags, but I think this is a bit "too magic" :dancing_women: .
I'm not super versed with this semantics/syntax so bare with me for awhile and correct me when I'm wrong

I had another look at this, I think it's too much magic for what it tries to accomplish :(, if I wanted to be safe on this I'd just add a variant for input/output and convert back/from to int. I'm not comfortable having to maintain this in the future.


Someway to refresh rewards status

The Issue
When we manually verify users, they have to refresh the app to get the new status. The user/me call only happens on app refresh, but some users are confused that closing/reopening the app does not refresh reward status (it doesn't if minimized to tray).  Can we have a refresh button on the rewards page somewhere?
System Configuration

LBRY Daemon version:
LBRY App version:
LBRY Installation ID:
Operating system:

Anything Else

Screenshots

Maybe just poll /user/me from that page or at least re-call it when navigated to?


Keep getting: "net::ERR_CONNECTION_TIMED_OUT"

Trying to follow this example (https://github.com/danialfarid/ng-file-upload/wiki/Direct-S3-upload-and-Node-signing-example) but I keep getting a connection timed out response back from AWS.
Here's my HTML:
<input type="file" ngf-select ngf-change="onFileSelect($files)">
Here's my JS:
$scope.onFileSelect = function(files) {

        if (files.length > 0) {
        var filename = files[0].name;
        var type = files[0].type;
        var query = {
            filename: filename,
            type: type
        };

        $http.put('/api/signing/', query)
            .success(function(result) {
                console.log(result);
                console.log(result.url)

                Upload.upload({
                    url: result.url, //s3Url
                    transformRequest: function(data, headersGetter) {
                        var headers = headersGetter();
                        delete headers.Authorization;
                        return data;
                    },
                    fields: result.fields, //credentials
                    method: 'PUT',
                    file: files[0]
                }).progress(function(evt) {
                    console.log('progress: ' + parseInt(100.0 * evt.loaded / evt.total));
                }).success(function(data, status, headers, config) {
                    // file is uploaded successfully
                    console.log('file ' + config.file.name + 'is uploaded successfully. Response: ' + data);
                }).error(function() {

                });
            })
            .error(function(data, status, headers, config) {

                console.log('data: ' + data + ' status: ' + status + ' headers: ' + headers);
                // called asynchronously if an error occurs
                // or server returns response with an error status.
            });
        }

};

Here's my Node slightly modified for my app's setup: (I've inserted my AWS ID, Key, and bucket info here directly while testing - will move to an ignore file once working);
var crypto = require('crypto');
var moment = require('moment');

var s3Url = 'https://XXXXXXXX.s3-website-us-east-1.amazonaws.com';

module.exports = function(app){

app.put('/api/signing', function(req, res){

        var request = req.body;
        var fileName = request.filename
        var path = '/' + fileName;

    var readType = 'private';

    var expiration = moment().add(5, 'm').toDate(); //15 minutes

    var s3Policy = {
        'expiration': expiration,
        'conditions': [{
                'bucket': 'XXXXXXXXX'
            },
            ['starts-with', '$key', path],
            {
                'acl': readType
            },
            {
              'success_action_status': '201'
            },
            ['starts-with', '$Content-Type', request.type],
            ['content-length-range', 2048, 10485760], //min and max
        ]
    };

    var stringPolicy = JSON.stringify(s3Policy);
    var base64Policy = new Buffer(stringPolicy, 'utf-8').toString('base64');

    // sign policy
    var signature = crypto.createHmac('sha1', 'XXXXXXXXXXXXXXXXXXXXXXXXXX')
        .update(new Buffer(base64Policy, 'utf-8')).digest('base64');

    var credentials = {
        url: s3Url,
        fields: {
            key: path,
            AWSAccessKeyId: 'XXXXXXXXXXXXXXXX',
            acl: readType,
            policy: base64Policy,
            signature: signature,
            'Content-Type': request.type,
            success_action_status: 201
        }
    };
     res.jsonp(credentials);
    });

};

Not sure if this is useful but when I try and do POST method I get connection refused.

Facing the same issue. @rikin311 were you able to solve it? thanks in advance.

@shi11 did you ever figure out a solution?


Added to find nth fibonacci number

A new file was not to be created here.
Please fix the existing code and make a new pull request


Patch HTTPClient to accept socket_connect_timeout

Monkey patch HTTPClient to accept socket_connect_timeout as optional configuration. If set, it is used as connect_timeout parameter when creating a TCPSocket.
Set the socket_connect_timeout in Diego::Client to (overall) connect_timeout / 2.
Add test to ensure that the httpclient version is not updated without revisiting the monkey patch.
Background information
:information_source: Use case: Mitigates the risk of Runner is unavailable and Stager is unavailable errors during a cf push when the client is unable to reach a bbs instance.

:x: First, discarded solution proposal: #3002

:heavy_check_mark: Second, merged solution proposal: #3048
:boom: Issues caused by #3048

#3109
#3112
sporadically failing sync-integration-tests

:information_source: Root cause: Net::HTTP and altered Diego::Client implementations are not thread-safe.
:arrows_counterclockwise: Revert PR: #3113

:heavy_check_mark: Third, merged solution proposal: #3170
:poop: Issues caused by #3170

cc-unit-tests failing (Concourse)

:information_source: Root cause: Concourse executes the unit tests with Ruby 3.1.3 (as opposed to the version 3.1.2 specified in .ruby-version) which includes a newer Net::HTTP version that reverted the usage of the Socket.tcp's connect_timeout option (see https://github.com/ruby/net-http/pull/74).
:arrows_counterclockwise: Revert PR: #3171

[x] I have reviewed the contributing guide

[x] I have viewed, signed, and submitted the Contributor License Agreement

[x] I have made this pull request to the main branch

[x] I have run all the unit tests using bundle exec rake

[ ] I have run CF Acceptance Tests

@johha and me prepared a local test environment to simulate an unresponsive bbs server and ran tests with different HTTP clients:

HTTP client
10 sequential pings
50 parallel threads
Notes

httpclient
:x: fails
n/a
current implementation

httpclient + monkey patch
:white_check_mark: succeeds
:white_check_mark: succeeds
set connect_timeout on TCPSocket

net_http (only v0.2.x)
:white_check_mark: succeeds
:x: fails

net_http_persistent(only with net_http v0.2.x)
:white_check_mark: succeeds
:white_check_mark: succeeds
pool_size: 50

net_http (other versions)
:x: fails
n/a

http.rb
:x: fails
n/a


myproject_ENABLE_ADDRESS_SANITIZER should be myproject_ENABLE_SANITIZER_ADDRESS

In this piece of code:
if(myproject_BUILD_FUZZ_TESTS)
  message(AUTHOR_WARNING "Building Fuzz Tests, using fuzzing sanitizer https://www.llvm.org/docs/LibFuzzer.html")
  if (NOT myproject_ENABLE_ADDRESS_SANITIZER AND NOT myproject_ENABLE_THREAD_SANITIZER)
    message(WARNING "You need asan or tsan enabled for meaningful fuzz testing")
  endif()
  add_subdirectory(fuzz_test)

endif()

could it be that myproject_ENABLE_ADDRESS_SANITIZER should be myproject_ENABLE_SANITIZER_ADDRESS and myproject_ENABLE_THREAD_SANITIZERshould be myproject_ENABLE_SANITIZER_THREAD` ?

In ProjectOptions.cmake, it is myproject_ENABLE_SANITIZER_ADDRESS.


Zero-sized batch causes operations like conv2d to crash (on GPU)

TensorFold fold seems to often generate zero-sized batch (an input batch with batch-size zero) in backward pass. Since many TensorFlow operations do not support using zero-sized batch, this behavior can cause crash on some operations, like conv2d_backward on GPU.
Below is an example to illustrate the conv2d crash (on GPU). Conditioning on "use_conv" key, the network dynamically decide whether to use conv2d on each instance. The session-run crash on backward (gradient) pass of conv2d using cudnn, causing a CUDNN_STATUS_BAD_PARAM error.
After debugging the code, I found that the conv2d backward pass crashes due to zero-sized batch (i.e. an input batch with batch-size zero) generated by fold. This issue is related to https://github.com/tensorflow/tensorflow/issues/5772
import numpy as np
import tensorflow as tf
import tensorflow_fold as td

def conv_layer(input, scope):
    with tf.device("/gpu:0"), tf.variable_scope(scope):
        filter = tf.get_variable("filter", shape=[3, 3, 3, 3])
    return tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1],
                        padding="SAME")

# Conditioned on "use_conv" key, decide whether to run convolution.
block = td.OneOf(td.GetItem("use_conv"),
                 [(True, td.GetItem("input") >> td.Tensor(shape=[10, 10, 3]) >> td.ScopedLayer(conv_layer)),
                  (False, td.GetItem("input") >> td.Tensor(shape=[10, 10, 3]) >> td.Identity())])

compiler = td.Compiler.create(block)
output = compiler.output_tensors[0]

# Some arbitrary optimization that involves conv2d backward
loss = tf.reduce_sum(output)
solver = tf.train.AdamOptimizer()
train_step = solver.minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())
data_1 = {"use_conv": True, "input": np.ones((10, 10, 3))}
data_2 = {"use_conv": False, "input": np.ones((10, 10, 3))}
# Crash on GPU with the following error.
# F tensorflow/stream_executor/cuda/cuda_dnn.cc:456]
#   could not set cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM
sess.run(train_step, feed_dict=compiler.build_feed_dict([data_1, data_2]))

My questions are:

Why would tensorflow fold generate some zero-sized batch on backward pass on the data above?
Is there any workaround to avoid putting some-zero batch in backward pass?

Thanks in advance!

TensorFlow fold instantiates each operation once in the graph, and then
evaluates it within a tf.while loop, which loops over the input tree by
depth.  If you don't use an operation at depth n, then the loop will still
evaluate the op at depth n, it just has batch size 0.  For example, a Tree
LSTM will call an embedding operation at depth 0 to embed all the leaves of
the tree, but it doesn't call embed at depth > 0 for the non-terminals of
the tree, so embed will be called with batch size 0 after the first
iteration of the loop.
The workaround is to use tf.cond to conditionally evaluate ops that aren't
happy with batch size 0.
On Sun, Feb 26, 2017 at 7:59 PM, Ronghang Hu<EMAIL_ADDRESS>wrote:

TensorFold fold seems to often generate zero-sized batch (an input batch
with batch-size zero) in backward pass. Since many TensorFlow operations do
not support using zero-sized batch, this behavior can cause crash on some
operations, like conv2d_backward on GPU.
Below is an example to illustrate the conv2d crash (on GPU). Conditioning
on "use_conv" key, the network dynamically decide whether to use conv2d on
each instance. The session-run crash on backward (gradient) pass of conv2d
using cudnn, causing a CUDNN_STATUS_BAD_PARAM error.
After debugging the code, I found that the conv2d backward pass crashes
due to zero-sized batch (i.e. an input batch with batch-size zero)
generated by fold. This issue is related to tensorflow/tensorflow#5772
https://github.com/tensorflow/tensorflow/issues/5772
import numpy as np
import tensorflow as tf
import tensorflow_fold as td
def conv_layer(input, scope):
with tf.device("/gpu:0"), tf.variable_scope(scope):
filter = tf.get_variable("filter", shape=[3, 3, 3, 3])
return tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1],
padding="SAME")
Conditioned on "use_conv" key, decide whether to run convolution.
block = td.OneOf(td.GetItem("use_conv"),
[(True, td.GetItem("input") >> td.Tensor(shape=[10, 10, 3]) >> td.ScopedLayer(conv_layer)),
(False, td.GetItem("input") >> td.Tensor(shape=[10, 10, 3]) >> td.Identity())])
compiler = td.Compiler.create(block)
output = compiler.output_tensors[0]
Some arbitrary optimization that involves conv2d backward
loss = tf.reduce_sum(output)
solver = tf.train.AdamOptimizer()
train_step = solver.minimize(loss)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
data_1 = {"use_conv": True, "input": np.ones((10, 10, 3))}
data_2 = {"use_conv": False, "input": np.ones((10, 10, 3))}
Crash on GPU with the following error.
F tensorflow/stream_executor/cuda/cuda_dnn.cc:456]
could not set cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM
sess.run(train_step, feed_dict=compiler.build_feed_dict([data_1, data_2]))
My questions are:

Why would tensorflow fold generate some zero-sized batch on
backward pass on the data above?
Is there any workaround to avoid putting some-zero batch in
backward pass?

Thanks in advance!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/fold/issues/23, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGGbTcGPTT3UDI2XUXPEIFn_ij3SiXnzks5rgkoogaJpZM4MMq3o
.

--
DeLesley Hutchins | Software Engineer |<EMAIL_ADDRESS>|<PHONE_NUMBER>

Thanks for the timely reply! I'll try the tf.cond workaround.

Hi, can you give an example how to use tf.cond in this case. For example, my code is below, can you show me how to use tf.cond? I'm new to tensorflow, so kindly help, appreciate it. Thank you!
def baseline_model():
inputs = Input(shape=input_shape)
reshape = Reshape(input_shape_channels)(inputs)
#normal convnet layer (have to do one initially to get 64 channels)
conv1 = Conv2D(50,(5,25),activation='tanh')(reshape)
do1 = Dropout(0.5)(conv1)
pool1 = MaxPooling2D(pool_size=(1,3))(do1)

conv2 = Conv2D(50,(3,5),activation='tanh')(pool1)
do2 = Dropout(0.5)(conv2)
pool2 = MaxPooling2D(pool_size=(1,3))(do2)

flattened = Flatten()(pool2)
fc1 = Dense(1000, activation='sigmoid')(flattened)
do3 = Dropout(0.5)(fc1)

fc2 = Dense(200, activation='sigmoid')(do3)
do4 = Dropout(0.5)(fc2)
outputs = Dense(note_range, activation='sigmoid')(do4)

model = Model(inputs=inputs, outputs=outputs)
return model


Add serial access for image debug in OCI

Currently the serial console is not enabled in the golden image.  This may prove to be useful for image deploy debugging.

A few references that will push this over the line:

https://www.virtualbox.org/wiki/Serial_redirect
https://gist.github.com/snb/284940/11e6354f170be602c9c2f67b59d489ed49ebd143
https://docs.oracle.com/en/virtualization/virtualbox/6.0/user/serialports.html

I also found this one:
http://notesofaprogrammer.blogspot.com/2020/05/enabling-serial-console-on-debian-linux.html
I still haven't validated that there is actually access via serial port, but I'm working on a PR that incorporates a console stty configuration, and also boots and cleans up the packer user password.


Model Validators are not adhered to when using validate_assignment

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
In instances where I set a model attribute to have some value (and validate_assignment is True, the resulting data does not adhere to the validation, instead continuing to assign the data.
Example Code
from __future__ import annotations

from typing import Any

from pydantic import BaseModel, ConfigDict, model_validator, ValidationInfo, field_validator

class DupController(BaseModel):

    model_config = ConfigDict(validate_assignment=True)

    field_1: list[str] = []
    field_2: list[str] = []

    @model_validator(mode="before")
    @classmethod
    def deduplicate_data(cls: type[DupController], data: dict[str, Any]):
        """Should always return empty lists, as a minimum example."""
        print("Validating")
        return {"field_1": [], "field_2": []}

obj = DupController()

print(obj)
# Should so empty lists

obj.field_1 = ["A"]

print(obj)
# Should so empty lists (but doesn't)

obj.field_2 = ["A"]

print(obj)# Should so empty lists (but doesn't)

Python, Pydantic & OS Version
pydantic version: 2.5.0
        pydantic-core version: 2.14.1
          pydantic-core build: profile=release pgo=true
                 install path: /Users/maphey1/Documents/projects/darc/api/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.4 (main, Jun 15 2023, 15:45:27) [Clang 14.0.3 (clang-14<IP_ADDRESS>.1)]
                     platform: macOS-13.6.3-arm64-arm-64bit
             related packages: fastapi-0.105.0 typing_extensions-4.9.0 email-validator-2.1.0.post1

@sammaphey,
Thanks for reporting this. This does look like a bug. @davidhewitt, when you return, could you take a look at this please?

Any updates on this @sydney-runkle ?

The problem seems to be that https://github.com/pydantic/pydantic-core/blob/1cefeae57609a876842b29c3ccba9df14aeae9ad/src/validators/model.rs#L184 is hard-coded to read from the model __dict__, but this happens after deduplicate_data has been called and is ignoring the result of deduplicate_data.
I think fixing this requires some rewiring / refactoring in pydantic-core though it doesn't seem trivial without a good chunk of time to sit down and think about the correct behaviour & abstractions.


Direct Install lacks options and cleanup

Thought I'd tag @akeif for this, since he's been very involved in getting the directinstall command working. Whilst the command works well, with only minor infidelities I thus far have blamed on Windows and haven't tracked down fully in debug, two points are worthy of immediate consideration: The command does not register common options, and folders are not deleted on failure or success.
For the former, force, all-yes, and destination are not available. Just like the rest of GPM, having them available cuts down on development-time significantly. For the latter, I noticed that regardless of whether the installation halts or succeeds, the temporary folders in /tmp/ are not deleted - at least none have been thus far.

Thanks @OleVik :smile:. I'm definitely interested in tackling this issue.
My understanding
If I get you right, here what should be done before solving this issue.

files created in /tmp/ while running directinstall should be removed
folders must be deleted whether the command succeed or fail
command force should be available and implemented properly
command all-yes should be available and implemented properly
command destination should be available and implemented properly

Request for more information on point 2
I get you are saying that this might happens only on Windows.
Also, what do you mean by folders?

Folders, as in directories. After running the command, whether it fails or not, it leaves behind a copy of the extracted extension in /tmp/Grav-randomhash/ for each attempt, but they are not subsequently removed by the process.
What happens on Windows is not related to point 1-5, they are merely current deficiencies of the implementation. On Windows two things happen on occasion: mkdir fails, or Composer's Autoload fails with the name is already in use in /tmp/Grav-randomhash/pluginname/pluginname.php. I suspect this is more an issue with Windows, though, and shouldn't be tackled unless it can be consistently reproduced.

All right. So I'll stick to this:

files created in /tmp/ should be removed whether the command succeed or fail
command force should be available and implemented properly
command all-yes should be available and implemented properly
command destination should be available and implemented properly

I'll take a look and make a plan before coding. I'd like create a merge request for each points but we'll see how this goes.
I'll try to do this tonight but will do Friday or Saturday at worst.

todo Add bin/gpm direct-install command to the API doc.

What the command does
Install

Grav
plugin (package)
theme (package)

from (.zip only?)

path
URL

Test packages links
Grav latest
Grav core 1.1.17
Grav update 1.1.17
Antimatter latest
Antimatter 2.1.0 zip
Antimatter 2.1.0 gz
Admin 1.1.17
Piwik 1.0.2

@OleVik In the bin/gpm direct-install context, what would be the difference between --yes-all and --force. To me, both would do exactly the same, so one would be an alias.

The aliases of force and yes-all are f and y, respectively. force will "Force re-fetching the data from remote", whilst yes-all "Assumes yes (or best approach) instead of prompting".
Essentially, you can force retrying/refreshing data from the GPM and/or pass the generic yes to all asked questions, such as "Dependencies are not met, proceed? y|n?". See InstallCommand.php.

Look likes theres two var I need to check out in system/src/Grav/Console/Gpm/DirectInstallCommand.php.
$tmp_zip and $tmp_source.

I don't think direct-install hit a cache. To what I see on GPM::downloadPackage and GPM::copyPackage in DirectInstallCommand.php it's copied and downloaded each time.
So the --force option will not be useful as of now.
Let's see what come out of this.

Installing an older Grav package delete the tmp folder and fails.
di https://github.com/getgrav/grav/releases/download/1.1.17/grav-v1.1.17.zip
#!/usr/bin/env php
Are you sure you want to direct-install https://github.com/getgrav/grav/releases/download/1.1.17/grav-v1.1.17.zip [y|N] y

Preparing to install https://github.com/getgrav/grav/releases/download/1.1.17/grav-v1.1.17.zip
  |- Downloading package...   100%
  |- Extracting package...    ok
  |- Checking destination...  ok
  |- Installing package...

  [RuntimeException]
  Cannot move non-existing folder.

direct-install <package-file>

Following my tests with packages from this comment successful installation clean temporary files.
See Folder::delete here and here.

I can't get my functional test to work. Instantiating a null output when I call the Grav command is giving me trouble.
This doesn't help me: http://stackoverflow.com/questions/18237519/symfony2-console-output-without-outputinterface
I get the error
[Symfony\Component\Console\Exception\InvalidArgumentException] The "package" argument does not exist.


Enhancement: Add a anvil option for Dark steel Helm to have the o2 and other sensor details added

Adding this enchant would allow the dark steel helm or any helm to have simalar functions to the space suit helmet!

Not happening, as it removes all incentives for the space suit over dark steel armor.


No auto-update of view on network share when creating or pasting files/folders

Description
I'm using an SMB network share, and when creating a folder, new file, or copy/pasting files and folders, the new element does not show up in the destination. Refreshing the view makes them appear.
I would like the new files to appear immediately, without having to refresh manually.
Steps To Reproduce
There's multiple ways, but here is one:

Copy some files from anywhare
go in an SMB network share location
Paste
Notice the view of the folder is still empty

Expected behavior
I'd like it to work just like on local drives. When I paste files in my SMB drive, they should show up where I just pasted them, and not have to refresh the view to see what I just did.
Files Version
Version: <IP_ADDRESS> OS Version: 10.0.19043.1645
Windows Version
Windows 10 Pro Version	21H1 Installed on	7/1/2021 OS build	19043.1645 Experience	Windows Feature Experience Pack 120.2212.4170.0
Relevant Assets/Logs
No need I think.

Thanks for the report. This is fixed in the latest preview version.


Image-based deduplication

Create component(s) that allow for images based deduplication.

see #476


Codeblocks in Markdown Preview don't use theme colors

VS Code version: Code 1.40.0 (86405ea23e3937316009fc27c9361deee66ffbf5, 2019-11-06T17:02:13.381Z)
OS version: Windows_NT x64 10.0.17763

Steps to Reproduce:

write markdown
open markdown preview
markdown preview does not use theme of code

Does this issue occur when all extensions are disabled?: Yes

As designed. We use highlight.js in the preview since many websites also use this for highlighting
Try this extension to have the preview's highlighting better match VS Code's colors: https://marketplace.visualstudio.com/items?itemName=bierner.markdown-shiki


Update replaceFirst() API with Ballerina Parser

Description:
$Subject

This task has been deprioritized now. Hence, closing this issue.
For more info: https://github.com/ballerina-platform/ballerina-standard-library/issues/2716#issuecomment-1063852430


allow pluses at the end of the device name regexp

https://github.com/keybase/keybase/pull/3904

need to update checkers test:
--- FAIL: TestCheckDeviceName (0.00s)
    checkers_test.go:59:
        	Error Trace:	checkers_test.go:59
        	Error:      	Not equal:
        	            	expected: true
        	            	actual  : false
        	Test:       	TestCheckDeviceName

also due to how device names are checked wrt normalization, it would still not be possible to have a device named e.g. "iPhoneXS" and then add a new one called "iPhoneXS+". But I think this is fine and desired.


Fixing Init test for Serial only

In Kokkos_Core_fwd.hpp we have
#if   defined( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_CUDA )
  typedef Cuda DefaultExecutionSpace;
#elif defined ( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_OPENMPTARGET )
  typedef Experimental::OpenMPTarget DefaultExecutionSpace ;
#elif defined( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_OPENMP )
  typedef OpenMP DefaultExecutionSpace;
#elif defined( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_THREADS )
  typedef Threads DefaultExecutionSpace;
//#elif defined( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_QTHREADS )
//  typedef Qthreads DefaultExecutionSpace;
#elif defined( KOKKOS_ENABLE_DEFAULT_DEVICE_TYPE_SERIAL )
  typedef Serial DefaultExecutionSpace;
#else
#  error "At least one of the following execution spaces must be defined in order to use Kokkos: Kokkos::Cuda, Kokkos::Experimental::OpenMPTarget, Kokkos::OpenMP, Kokkos::Threads, Kokkos::Qthreads, or Kokkos::Serial."
#endif

Thus, the only way Serial is the default execution space is when all other spaces are disabled. So it is fine to leave the current code in DTK_Core.cpp.

Codecov Report

Merging #260 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #260      +/-   ##
==========================================
- Coverage   96.94%   96.94%   -0.01%
==========================================
  Files          49       49
  Lines        2491     2485       -6
==========================================
- Hits         2415     2409       -6
  Misses         76       76

Impacted Files
Coverage Δ

packages/Interface/test/tstInit.cpp
92.1% <100%> (-1.08%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9492a5...023150d. Read the comment docs.

In gdb, it looks like there is a problem with getting typeid(default_space) (no symbol for default_space and no symbol for DefaultExecutionSpace in Kokkos

This is a red herring. You are right in the sense that gdb does not show those symbols. But if you add printing typeid(default_space).name() in the actual code, you can see it.
The logic in the PR was flawed due to my stupidity. I tried to make the code easy to read, and moved the code from main() to a separate function. That was a mistake, as you need to take negative of only one clause in that function.
I updated the PR with the right logic, and tested it on Serial. Should work for you too.


chore(main): release 1.4.4

:robot: I have created a release beep boop
1.4.4 (2024-03-26)
Bug Fixes

main: Add favicon ico file (b3b99fe)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/CoreySpohn/lod_unit/releases/tag/v1.4.4 :sunflower:


error in package install

I am trying to install some emacs packages, but I got two errors, melpa.org package not found and melpa.org temporary failure in name resolution

The latter is presumably because your network was not able to connect to MELPA (I'm not aware we had any downtime...). The former is either a result of that same issue, or because your package list is out of date: try M-x package-refresh-contents.


Require that server copy the room type when upgrading rooms

Suggestion
If I understand correctly, Element Web doesn't allow upgrading spaces as there is no mechanism for them to know that the homeserver will perform the operation correctly (without losing the information that the room is a space).

https://github.com/vector-im/element-web/issues/19208

It looks like this may get resolved by #3385 and I think this also prevents upgrading #3588 rooms (or at least I am yet to get told otherwise in a week)

Note that Synapse actually does this already: matrix-org/synapse#10774

ftr, the spec currently makes it an implementation detail for the field to carry over - open to suggestions (meaning MSCs) which discuss a different approach.


improvement regex

improvement regex with compile flag

@EngRajabi I am not sure if that makes any good here.

Compiled regular expressions maximize run-time performance at the expense of initialization time.

Can you provide some bench marking to support your idea please?

Yes, I did.It is a bit slow to use the first time, but much faster the next time
https://medium.com/@mohsen_rajabi/how-to-write-a-regex-very-fast-in-c-best-practice-875d386c0485

Yes, I did.It is a bit slow to use the first time, but much faster the next time

@DavoudEshtehari and @Wraith2 any opinion on this? I think we should not sacrifice initialization time for runtime performance unless we have a good benchmark to show a practical test case.
@EngRajabi can you write a test case with SqlClient  and do some benchmarking on it? The one you have provided is merely for RegEx and does not contain any SqlClient related tests.

@DavoudEshtehari and @Wraith2 any opinion on this? I think we should not sacrifice initialization time for runtime performance unless we have a good benchmark to show a practical test case.

I looked at this when it was posted and my opinon was that it is not harmful. It's not in a heavily used code path for any performance benchmarks I've seen so it's not going to cause trouble for most people. If it is heavily used for a small number of people then this is probably an improvement for them. Ideally I'd like to see benchmark numbers but it's a single line change in a low traffic area that might benefit users so I'm not against it.
We keep a reference the to the regex so we're not relying on the static regex cache we will do the work only once. On netfx compiled regex output an assembly which is going to cause codegen which is slow but if it's heavily used there may be an overall perf benefit. On netcore <7 compiled flag does nothing, on 7+ it may cause a build time code generator to generate the compiled expression which will be faster for users in all cases.

Following the Wraith2 comment, it'll improve a small number of customers with high loads.

Regular Expression Performance

Best Practices
To maximize the performance of regular expressions in the .NET Framework, we recommend that you do the following when working with regular expression objects:

Call static regular expression matching methods instead of repeatedly instantiating the same regular expression object.
Use interpreted regular expressions when you expect to call pattern-matching methods a limited number of times to parse text.
Use compiled regular expressions to optimize performance for regular expression patterns that are known in advance and when the frequency of calls to the regular expression’s pattern-matching methods can vary extensively.

Thanks for the explanation friends
According to Microsoft documentation
https://docs.microsoft.com/en-us/dotnet/standard/base-types/best-practices
And the test I took
https://medium.com/@mohsen_rajabi/how-to-write-a-regex-very-fast-in-c-best-practice-875d386c0485
The following should be noted in regular expressions
-Use the Regex object statically because they are threadsafe
-Use the compiled flag


which directories should i persist in emqx

Question

which directories should i persist in emqx ?
Description
I need to persist each application credential i create in emqx in case of backup or something goes wrong.if s.th goes worng with my docker container, my software could still be available and therefore i don't have to create another app id and app secret.
i already tried to persist like below :

./data/emqx/data:/opt/emqx/data/

but unfortunately it does not work as it expected. when i docker down and docker up again my app credentials will be lost but i believe the logs and data are there.
overall which directories should i persist ? especially if i need to activate emqx_retainer and save messages to disk

/opt/emqx/log
/opt/emqx/data
/opt/emqx/lib
/opt/emqx/etc

i added these volumes in my docker-compose file and created an application called "johndoe" in emqx dashboard, but unfortunately this just does not work.
i ran the following command :

docker-compose up -d --force-recreate emqx

but when i visited dashboard, application was not there and it seemed it was deleted.
i searched in these directories and i could find this file in the directory
i ran this command in my host machine and i am pretty sure this file exists in both host and container.

grep -nri johndoe .
Binary file<EMAIL_ADDRESS>matches

services:
  emqx:
    stdin_open: true
    tty: true
    restart: always
    image: emqx/emqx:v3.0-rc.5
    container_name: emqx
    entrypoint: []
    networks:
      - confluent
    ports:
      - "18083:18083"
      - "1883:1883"
      - "8080:8080"
    volumes:
      - ./emqx/log:/opt/emqx/log #working
      - ./emqx/data/mnesia:/opt/emqx/data/mnesia #working
      - ./emqx/lib:/opt/emqx/lib #not working
      - ./emqx/etc:/opt/emqx/etc/plugins/ #not working

Using the above configuration i was not able to start the docker container
Logs:
Creating emqx ... done
Attaching to emqx
emqx                  | node.max_ports=1048576
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | listener.tcp.external.acceptors=64
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | listener.ssl.external.acceptors=32
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | node.process_limit=2097152
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | node.max_ets_tables=2097152
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | listener.ws.external.acceptors=16
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  |<EMAIL_ADDRESS>emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory
emqx                  | run_erl[510]: Erlang closed the connection.
emqx                  | 2018-12-11T09:37:06.443197+00:00 [error] Translation for 'emqx_auth_pgsql.server' expected to find setting 'auth.pgsql.database' but was missing
emqx                  | 2018-12-11T09:37:06.443349+00:00 [error] Translation for 'emqx_auth_mongo.auth_query' expected to find setting 'auth.mongo.auth_query.password_hash' but was missing
emqx                  | 2018-12-11T09:37:06.443503+00:00 [error] Translation for 'emqx_auth_http.auth_req' expected to find setting 'auth.http.auth_req.params' but was missing
emqx                  |
emqx                  | =====
emqx                  | ===== LOGGING STARTED Tue Dec 11 09:41:03 UTC 2018
emqx                  | =====
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:21Z']:waiting emqx
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:22Z']:waiting emqx
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:24Z']:waiting emqx
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:25Z']:waiting emqx
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,file_io_server,g
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:26Z']:waiting emqx
emqx                  | init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv
emqx                  |
emqx                  | Crash dump is being written to: erl_crash.dump...done
emqx                  | ['2018-12-11T09:41:28Z']:waiting emqx
emqx                  | ['2018-12-11T09:41:28Z']:timeout error
emqx exited with code 1

I have created the emqx/log, emqx/data, emqx/lib, emqx/etc in my local directory where docker-compose.yml resides.
Using only the log and mnesia seems to start the container.
I want to persist my app configuration in case of container restarts (i.e. when container is recreated)
What is the right way to achieve this?
Thanks!

@deepaksood619, I faced the same issue as you regarding persistence (https://github.com/emqx/emqx-docker/pull/91#issuecomment-445826906) ... unfortunately no final solution yet.
I haven't tried yet, but what would be a workaround is:

start a docker container without given persitstence directories, but instead persist /tmp
docker run -it -v /hostdir:/tmp --name=temp-emqx emqx/emqx:v3.0-rc.5
login to the container
docker exec -it  temp-emq /bin/sh
within the container copy the required dirs (/opt/emqx/log,/opt/emqx/data,/opt/emqx/lib and /opt/emqx/etc/plugins/) to /tmp. Maybe the emqx process needs to be stopped before copying.
cp -Rv /opt/emqx/log /opt/emqx/data /opt/emqx/lib /opt/emqx/etc/plugins /tmp
Stop and remove container. At this point all dirs should be available on your host
Start your docker service / container pointing to the host dirs which where copied in previous step.

@RaymondMouthaan, i tried this approach but was not able to copy the files as both emqx and root user with both emqx started and stopped.
deepaksood@Deepaks-MacBook-Pro  ~  docker exec -it --user root temp-emqx /bin/sh
/opt/emqx #
/opt/emqx # ./bin/emqx stop
ok
/opt/emqx # cp -Rv /opt/emqx/log /opt/emqx/data /opt/emqx/lib /opt/
emqx/etc/plugins /tmp
cp: can't create directory '/tmp/log': Permission denied
cp: can't create directory '/tmp/data': Permission denied
cp: can't create directory '/tmp/lib': Permission denied
cp: can't create directory '/tmp/plugins': Permission denied
/opt/emqx # whoami
root

Same i tried as emqx user too, but with no success.

@deepaksood619 can you do:
`ls -al /' within the container? ...

@RaymondMouthaan
/ # ls -al
total 60
drwxr-xr-x    1 root     root          4096 Dec 11 15:40 .
drwxr-xr-x    1 root     root          4096 Dec 11 15:40 ..
-rwxr-xr-x    1 root     root             0 Dec 11 15:40 .dockerenv
drwxr-xr-x    1 root     root          4096 Dec  1 10:16 bin
drwxr-xr-x    5 root     root           360 Dec 11 15:40 dev
drwxr-xr-x    1 root     root          4096 Dec 11 15:40 etc
drwxr-xr-x    1 root     root          4096 Dec  1 10:17 home
drwxr-xr-x    1 root     root          4096 Dec  1 10:16 lib
drwxr-xr-x    5 root     root          4096 Sep 11 20:23 media
drwxr-xr-x    2 root     root          4096 Sep 11 20:23 mnt
drwxr-xr-x    1 root     root          4096 Dec  1 10:17 opt
dr-xr-xr-x  205 root     root             0 Dec 11 15:40 proc
drwx------    1 root     root          4096 Dec 11 15:40 root
drwxr-xr-x    2 root     root          4096 Sep 11 20:23 run
drwxr-xr-x    2 root     root          4096 Sep 11 20:23 sbin
drwxr-xr-x    2 root     root          4096 Sep 11 20:23 srv
dr-xr-xr-x   13 root     root             0 Dec 11 15:40 sys
drwxr-xr-x    2 root     root            64 Dec 11 14:57 tmp
drwxr-xr-x    1 root     root          4096 Dec  1 10:16 usr
drwxr-xr-x    1 root     root          4096 Dec  1 10:16 var
/ # whoami
root
/ #

hmm, weird ..
have you tried, creating a dir within /tmp like this?
mkdir -p /tmp/test

already done but with no success
/ # whoami
root
/ # cd tmp/
/tmp # mkdir -p test
mkdir: can't create directory 'test': Permission denied
/tmp #

Also tried less /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
emqx:x:1000:1000:Linux User,,,:/home/emqx:

But can't seem to find the problem

@RaymondMouthaan, here is the DockerFile emqtt is using for building it's images. Please have a look.
https://github.com/emqx/emqx-docker/blob/emqx30/Dockerfile

Like you, i am confused as well .. can you try without persist ? (assuming you have used -v in your docker command)?

Yes i am able to create the directory now inside /tmp without the volume mount

Great, then there could be at least two reasons I can think of why it doesn't work with /tmp

permissions issues from within the docker container to the host dir, meaning root in the docker container has not the permissions to write the host dir
or because /tmp is not "marked" as volume in the Dockerfile

What you could test is .. mount volume as before but add a file to the host dir and check if this file is visible inside the container.
Another test/solution would be, mount /opt/emqx/log to your host file dir, then copy all other dirs to that. So basically instead of /tmp use /opt/emqx/log instead. Maybe remove created logs before copying...

Try ./hostdir instead /hostdir and make sure the dir exists.

Yes! it worked, data is copied to hostdir and emqtt is starting. But the initial problem persists, that an app created from management console in localhost:18083 doesn't persist after new container recreate.
This is what i did-
docker run -it -v /Users/deepaksood/hostdir/data:/opt/emqx/data -v /Users/deepaksood/hostdir/lib:/opt/emqx/lib -v /Users/deepaksood/hostdir/plugins:/opt/emqx/etc/plugins -p 18083:18083 -p 8080:8080 --name=temp-emqx emqx/emqx:v3.0-rc.5

All the previous session data is there in the /Users/deepaksood/hostdir/data folder but after new docker container new data folders are created.
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls -al data/mnesia/
total 0
drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:06 ./
drwxr-xr-x  6 deepaksood  staff   192B Dec 12 00:01 ../
drwxr-xr-x  7 deepaksood  staff   224B Dec 12 00:06<EMAIL_ADDRESS>drwxr-xr-x  7 deepaksood  staff   224B Dec 12 00:01<EMAIL_ADDRESS> deepaksood@Deepaks-MacBook-Pro  ~/hostdir 
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls
data/    lib/     plugins/
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir 
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir 
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls data
configs/        emqx_erl_pipes/ loaded_plugins  mnesia/
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls -al data
total 8
drwxr-xr-x  6 deepaksood  staff   192B Dec 12 00:01 ./
drwxr-xr-x  5 deepaksood  staff   160B Dec 12 00:01 ../
drwxr-xr-x  8 deepaksood  staff   256B Dec 12 00:06 configs/
drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:06 emqx_erl_pipes/
-rw-r--r--  1 deepaksood  staff    60B Dec 12 00:01 loaded_plugins
drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:06 mnesia/
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls -al data/emqx_erl_pipes
total 0
drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:06 ./
drwxr-xr-x  6 deepaksood  staff   192B Dec 12 00:01 ../
drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:06<EMAIL_ADDRESS>drwxr-xr-x  4 deepaksood  staff   128B Dec 12 00:01<EMAIL_ADDRESS> deepaksood@Deepaks-MacBook-Pro  ~/hostdir  ls -al data/configs
total 400
drwxr-xr-x  8 deepaksood  staff   256B Dec 12 00:06 ./
drwxr-xr-x  6 deepaksood  staff   192B Dec 12 00:01 ../
-rw-r--r--  1 deepaksood  staff    77K Dec 12 00:01 app.2<IP_ADDRESS>.29.22.conf
-rw-r--r--  1 deepaksood  staff    14K Dec 12 00:01 app.2<IP_ADDRESS>.29.23.config
-rw-r--r--  1 deepaksood  staff    77K Dec 12 00:06 app.2<IP_ADDRESS>.36.47.conf
-rw-r--r--  1 deepaksood  staff    14K Dec 12 00:06 app.2<IP_ADDRESS>.36.48.config
-rw-r--r--  1 deepaksood  staff   241B Dec 12 00:01 vm.2<IP_ADDRESS>.29.23.args
-rw-r--r--  1 deepaksood  staff   241B Dec 12 00:06 vm.2<IP_ADDRESS>.36.48.args
 deepaksood@Deepaks-MacBook-Pro  ~/hostdir 

How can i make emqtt use old data folders instead of creating new ones.

Okee, this confuses me a lot, basically when the dirs / files exists on the host, then emqx creates new ones.. If they don't exist, emqx won't even start 🤔

Yes that sums it all... 😐
On Wed 12 Dec, 2018, 12:27 AM Raymond Mouthaan<EMAIL_ADDRESS>wrote:

Okee, this confuses me a lot, basically when the dirs / files exists on
the host, then emqx creates new ones.. If they don't exist, emqx won't even
start 🤔
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/emqx/emqx-docker/issues/89#issuecomment-446319252,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMmEVEv6tK19L7mFyEjGMdlTRRRBEYw5ks5u4AAegaJpZM4YxbLJ
.

Any update on this issue?

As far as I can tell emqx is creating folders to contain config files based on the id of the node. This is probably in cases where there are multiple nodes running on a single physical machine without the use of containers in order to avoid conflicts when reading and writing to these files; When a new container is created, the hostname and possibly IP address also change, changing the name of the emqx node and therefore the folders that the node writes to.
The solution that worked for me was to manually set the emqx node name via an environment variable to  ensure that the node name and therefore configuration folder being used was consistent across container destruction and recreation. The environment variable for controlling this is EMQX_NODE__NAME and should contain data in the format of EMQX_NAME@EMQX_HOST, in my case this was<EMAIL_ADDRESS>and user login, and applications are now persistent.

@WhyKickAmooCow thanks for your reply! Could you supply an example of you full docker command / docker compose file, how you manage to get persistence to work?

Here is a sample of my docker-compose file. For me it really was as simple as keeping the suggested  folders above in volumes, and setting an environment variable.
yaml     mqtt:         image: emqx-docker-v3.0.0         environment:             -<EMAIL_ADDRESS>       ports:             - 1883:1883             - 8083:8083             - 8084:8084             - 8080:8080             - 18083:18083         volumes:             - ./emqx-config:/opt/emqx/etc             - ./emqx-data:/opt/emqx/data             - ./emqx-lib:/opt/emqx/lib         networks:              ash:                 aliases:                     - mqtt.local

Probably my mistake but I still can't get it to work with:
services:
  emqx-master:
    image: raymondmm/emqx
    hostname: emqx-master
    environment:
      - TZ=Europe/Amsterdam
      - EMQX_NAME=emq
      - EMQX_HOST=master.mq.tt
      - EMQX_NODE__COOKIE=ef16498f66804df1cc6172f6996d5492
      - EMQX_WAIT_TIME=60
      -<EMAIL_ADDRESS>    networks:
      indonesia-net:
        aliases:
          - master.mq.tt
      proxy_indonesia-net:
    volumes:
      - /shares/rpi-cluster-1/emqx-master/log:/opt/emqx/log
      - /shares/rpi-cluster-1/emqx-master/data:/opt/emqx/data
      - /shares/rpi-cluster-1/emqx-master/lib:/opt/emqx/lib
      - /shares/rpi-cluster-1/emqx-master/etc:/opt/emqx/etc

These are the errors I am getting:
cat: can't open '/opt/emqx/etc/emqx.conf': No such file or directory

cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory

cat: can't open '/opt/emqx/etc/emqx.conf': No such file or directory

cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory

cat: can't open '/opt/emqx/etc/emqx.conf': No such file or directory

cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory

cat: can't open '/opt/emqx/etc/emqx.conf': No such file or directory

cat: can't open '/opt/emqx/etc/plugins/*': No such file or directory

vm.args needs to have a -name parameter.

  -sname is not supported.

=====

===== LOGGING STARTED Thu Jan  3 02:16:07 CET 2019

=====

init terminating in do_boot ({load_failed,[supervisor,proc_lib,lists,logger_simple_h,logger_config,logger_backend,logger_server,logger_filters,logger,kernel,heart,gen_server,gen_event,gen,file_io_serv

Crash dump is being written to: erl_crash.dump...done

/opt/emqx/bin/emqx_ctl: line 40: echoerr: not found

['2019-01-03T01:18:54Z']:waiting emqx

/opt/emqx/bin/emqx_ctl: line 40: echoerr: not found

['2019-01-03T01:18:55Z']:waiting emqx

/opt/emqx/bin/emqx_ctl: line 40: echoerr: not found

['2019-01-03T01:18:56Z']:waiting emqx

/opt/emqx/bin/emqx_ctl: line 40: echoerr: not found

On first setup I let the container run without the lib and etc volumes mounted so not to overwrite the existing folders in the container with empty ones. Then I used docker cp container_id:folder src_dir to copy the lib and etc dirs from the running container into my volume. I could then get the docker-compose run without any errors, although you might be alright to only copy the etc folder given your errors.

Thanks, that got me a little further... Still got some issues, but I'll try again later. Either way it's not very user friendly to persist emqx. Hopefully emqx team will improve this, also when running in clustered setup. A full example with a master and a worker including persistence for both would very helpful 👍🏽

This here is really an issue when deploying emqx. It does cost time to figure it all out. I finally got really far but now it keeps telling me this remaining bit and then exists abnormally:
Error opening file: /opt/emqx/data/configs/app.2<IP_ADDRESS>.10.33.conf no such file or directory

The logic should be:

use container default files as default if host folders are empty
host changes map to container (by default)

I have resolved persist with emqx/emqx:v4.0.0, but I think it is also valid for previous version.
First, use yml file without volume. You must set NODE_NAME and NODE_HOST to make node unique. Following is docker-compose.yml:
version: "3"
services:
    emqx:
        container_name: emqx
        image: emqx/emqx:v4.0.0
        networks:
            cloud:
                aliases:
                    - emqx
        environment:
            - EMQX_NAME=mqtt
            - EMQX_HOST=<IP_ADDRESS>
        ports:
            - "1883:1883"
networks:
    cloud:
        external: true

After startup the container successfully, you should copy the folder in container to your persist location:
docker cp emqx:/opt/emqx/etc ~/emqx/etc
docker cp emqx:/opt/emqx/lib ~/emqx/lib
docker cp emqx:/opt/emqx/data ~/emqx/data
docker cp emqx:/opt/emqx/log ~/emqx/log

Then change the owner, group of these folders:
cd ~/emqx
chown -R 1000:1000 data etc lib log
chmod -R 775 data etc lib log

Modify the docker-compose.yml and add the volume mapping:
volumes:
            - $HOME/emqx/lib:/opt/emqx/lib
            - $HOME/emqx/etc:/opt/emqx/etc
            - $HOME/emqx/data:/opt/emqx/data
            - $HOME/emqx/log:/opt/emqx/log

Remove the container and recreate one. That is OK.


_settings.scss file is not being generated with build

The auto-generated _settings.scss file is not being generated with the build

@gakimball any idea how to get this working with octophant 1.0?  I'm getting the following error when I updated:
Parser for annotation BaseApps not found
I'm not sure where to define the required information

@gakimball I figured it out, just need to manually update the files to the new format

You need to use<EMAIL_ADDRESS>to parse settings variables in that format. The older settings parser was manually parsing strings out of a Sass file. The newer version uses SassDoc.
Here's the readme for the older settings parser. You need to change the delimiter from the default @Foundation.settings, so I think the setup is something like:
const octophant = return require('octophant');

gulp.task('settings', function() {
  octophant(['scss/components/*.scss'], {
    start: '/// @BaseApps.settings'
  });
});

Here's what the newer formatting looks like. We started that with Foundation 6 (and Foundation for Emails 2), and that's the format that<EMAIL_ADDRESS>can process.

@gakimball do you know if there is a way to get octophant v1.0 to produce the same html partials that it did for 0.1.0?  I like how the docs are configured to autoupdate the SCSS settings sections for each component and would like to get that working again.
Perhaps add a callback for each SCSS group being parsed so that those settings can be written to another file?  To make it more generic, possible provide a transform function so that the settings can be manipulated if desired.  If you just want to do something with the settings (like print them to a file), the transform function can just be a pass-thru.

I could, but I'm wary of continuing work on any of the libraries I worked on while I was at ZURB. I don't have much time in general, and I'm not supposed to be a ZURB admin still, they just haven't remembered to remove me yet.
That partials feature was removed because, for Foundation 6, we developed a more fully-formed solution for documentation: a tool called Supercollider. It's built for sites like the Foundation docs, which are more component-driven and implement documentation from multiple programming languages.
Now, setting it up takes a bit of work—right now, it has no default template (because we custom wrote one).
However, I've forked Supercollider and intend to expand on it, because I think others could find it useful if it was easier to use out of the box. I almost have the Sass template ready, at which point I'll have something you can use.
If you have all of the Sass variables commented SassDoc-style (with three slashes and a @type annotation), we can recreate the same template functionality, and also possibly expand it into Angular. One plugin would be added to your build system, and you'd have a table of Sass variables appended to the bottom of each page with Sass.
return gulp.src(paths.html.templates)
    // update version throughout framework
    .pipe($.replace(/<version>/g, version))
    .pipe(spacedoc({
      adapters: ['sassdoc'],
      markdown: false,
      keepFm: true
    })
    .pipe(routes({
      path: 'build/assets/js/routes.js',
      root: 'docs'
    }))
    .pipe(gulp.dest('./build/templates'))
  ;

@gakimball sounds great!  I'll manually update the SCSS templates for now until we can start using an automated solution.


Show in File Manager Clarification

Changed the context menu 'Show in File Manager' action description to better reflect actual behavior. Fixes #20897
PS. I'm not sure if I need to do something to add a new translatable string or if it's picked up automatically by the translation system.

Apart from my nitpicks, the proposed changes seem fine.

I think I cleaned up everything.

Thanks, looks good!


Multi-delete support

I can't remember why exactly we needed this in our internal fork, but we did, so here it is!
I have also redirected the log in the testing output into a temporary file as I was having trouble seeing which of my tests was failing after I added this. I would like at some point to provide an interface to allow users of the GoFakeS3 package to be able to substitute the logging, but I didn't want to overload this PR any more than it already was.
An interesting thing that came up is that our internal fork started using the s3 messages directly from the Go AWS SDK, which I think I'd like us to avoid doing here - if people use this package in their tests, they may have the SDK v1 or the SDK v2 included in their project already; I think we should stay agnostic and not use the SDK directly ourselves (except in tests).

Codecov Report

Merging #19 into master will decrease coverage by 2.81%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master     #19      +/-   ##
=========================================
- Coverage   58.61%   55.8%   -2.82%
=========================================
  Files          10      10
  Lines         476     500      +24
=========================================
  Hits          279     279
- Misses        176     200      +24
  Partials       21      21

Impacted Files
Coverage Δ

messages.go
64.28% <ø> (ø)
:arrow_up:

gofakes3.go
48.06% <0%> (-4.77%)
:arrow_down:

routing.go
64% <0%> (-4.09%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e05d68...d750e4f. Read the comment docs.

I think we should stay agnostic and not use the SDK directly ourselves

Yes, the main code base should stay as lean and clean as possible.

Hahaha yeah the codecov thing did a good job of encouraging me to make the extra effort. I've been picking over the reports and noticed there's some things I expected to show up as covered that weren't, so I might need to go back over a few things!


HoloLens 2 Emulator December 2019 Update

Docs Build status updates of commit ff55d41:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

mixed-reality-docs/hololens-emulator-archive.md
:white_check_mark:Succeeded
View

mixed-reality-docs/install-the-tools.md
:white_check_mark:Succeeded
View

mixed-reality-docs/using-the-hololens-emulator.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


improve docker examples

Summary of changes
Changes introduced in this pull request:

used safer env variable expansion instead of ~. It seems the latter not supported for mounting volumes in some docker (or shell?) versions. Thanks @lemmih
added an example using host db

Other information and links

I guess rust 1.62 is out with new lints. :)


Fix typo in highlight name

In commit view diff deletion highlighting wasn't working due to a typo
The correct highlight name defined in lua/neogit/lib/hl.lua is NeogitDiffDeletions

Good catch, thanks


Typescript Branch

Hi there. Recently I've started to push forward typings here: https://github.com/thisdotvoid/adonisjs-types. However, two things bother me:

if we use "use" function,  we have to make monkey patching adding export{}  otherwise TS will throw "cannot redeclare block-scoped variable" . If we use es6 import or require() , then we just lose code completion, because we including .js file. And Typescript's allowJs works bad if .js lives in node_modules. There is so much more to write about this, I've tried so many ways,  but none of them is elegant and working at the same time.
typings is an additional maintenance overhead - once source changed, typings are obsolete/broken and must be changed too.

After fighting for long hours for few days, for me there is only option that left: source code of Adonis must be written in TS, as files with .ts extension. I,ve already started to try this locally,  and it works very well.
Now what I am going to do: fork, transform js extensions to ts,  write required ts types,  class fields, etc. And write some doc section about typescript. What is important, Adonis will still be usable by those who do not know/use/like typescript.
Now my question: @thetutlage is it ok? What can you say before I start? Hopefully you just say OK, but that OK should mean that you have made thoughtful decision. Thanks

Hi @anurbol 👋
Code-base is currently being rewritten for V5 in TypeScript.

@anurbol Seems like you have spent some good time researching on how to add intellisense in AdonisJs. I did same and found that Valid Javascript is Typescript is not 100% true.
I decided to re-write bunch of stuff in Typescript and our reasons can be different. For me Intellisense or atleast VsCode is not important than the ease of using AdonisJs.
What I am not doing

I want, Adonis to work like a charm with Javascript. However, you can opt in to use Typescript.
The standard modules of Node.js should work fine, without any transpilation.
I am not making changes in the public API of Adonis, just to make Typescript happy.

Why I am using Typescript

For automated API docs via Typedoc.
Improve intellisense, if I can without sacrificing the charm of Adonis.

Thanks for the quick answers!
@RomainLanz, that's a good news! However, I failed to find corresponding branch or commits. If it's currently being rewritten, then where it is happening? I mean, is it done locally? Is the job just at the very-very beginning, so there is no single commit yet? Or is it now just a stage of discussion, if yes then where (I haven't found one on github and gitter)? Or is it just in our heads still  :D
@thetutlage
"I decided to re-write bunch of stuff in Typescript" - Same questions as above, is it just decision, that only exist at the moment? Which is also totally makes me happy, because that decision is key thing, other is matter of time/code. I even tracked your activity, and I noticed you are currently working on Edge template system, however using TypeScript, that's great. If you are going to start re-writing main library, then when it is going to happen? If not soon, can I start it now (in fork), without fear that you or someone already has done some fraction of work (e.g on your computer) and we'd have conflicts? If fork is not acceptable, maybe branch is a way to go (better, because others will see it and participate)?
I totally agree with you on all points, and happy that you keep exactly all mentioned principles, I just was slightly confused by this sentence, and not sure what you mean: "For me Intellisense or atleast VsCode is not important than the ease of using AdonisJs" - ease of using AdonisJS for me (and maybe others) definitely is a good code completion (intellisense), instead of learning docs/API. I really feel old and tired of hundreds of documentations that I have to read again and again. I want to give you good example of reason of my motivation :
I started to migrate my migrations (sorry my wordplay) from Laravel to Adonis. I see new API, new method names like notNullable() instead of null(false), yet main principles are the same. Reading docs, or even diving into source code is exhausting, I just want to migrate as quick as possible and continue developing business logic. Instead of investigating API/docs, I've installed typings and got code completion, so I quickly (seconds rather than minutes) see that new method is notNullable. Also, chance of mis-typing is next to none, I see which methods are chainable and what they return, etc. My long-term memory and mental capacity freed for business logic and other tasks, which is key for effectiveness. I hope you include this case in concept of "ease of using".
All the other points, I totally agree with them:
1. I want, Adonis to work like a charm with Javascript. However, you can opt in to use Typescript.
Yes, only thing that'll change, that source files would contain not only .js files but also .ts (but .js files would be compiled, non-human-readable)
2. The standard modules of Node.js should work fine, without any transpilation.
I see no single reason why they'd break. No transpilation (e.g. babel, webpack, i do not love them frankly) would be used, only TypeScript compilation .ts => .js
3. I am not making changes in the public API of Adonis, just to make Typescript happy.
No one single change. Again, I see no reasons why this can happen.

Found this: https://github.com/adonisjs/discussion/issues/62 but i hardly answered my questions. Please let me start work on TS ASAP, if there is no conflicting current work, because I do not even want touch things in bare JS (I have physical headaches from it)

@anurbol
Yes, no public work has started with Typescript on AdonisJs. I started with Edge to see how the tooling works and how helpful Typescript will be for the end user.
For me Intellisense or atleast VsCode is not important than the ease of using AdonisJs
By this I mean, Intellisense doesn't work 100% when your project is written in Javascript, even though type definitions exists. For example
const Route = use('Route')

I don't think in a Javascript project, the Intellisense will be able to guest the type of use('Route').
Also what makes Javascript great is it's dynamic nature. So I have to give up all the dynamic stuff, just to make Vscode happy, then ofcourse I am not doing it.
For example: You can add macros to Request and Response classes. The macros are added at runtime
Request.macro('username', function () {
  return this.input('username')
})

and you can use the macro as any other method on the Request object. So again I am not sure if Intellisene will be able to detect username as a method on Request class.

I don't think in a Javascript project, the Intellisense will be able to guess the type of use('Route')
:D Easy! In fact it already works with https://github.com/anurbol/adonisjs-types/blob/develop/index.d.ts (scroll to the very end and you'll understand how it works)
Even your example with Request.macro is handle-able with TS same way:
`interface Request { static macro(what:'username', callback: () => InputClassOrWhatever) }
`

Well, from your answer I concluded that there is no work started. That means I can start it by forking? What you think about TypeScript branch? I really want to complete this work in a week. Maybe 2 days if I will have luck. Maybe 2 weeks though, if not.

Good! Let's begin :-) I'll try my best: https://github.com/anurbol/adonis-framework/tree/typescript

Nope, it will work as it does right now

@PazzaVlad no, absolutely nothing changes in API (no one gonna notice any changes or change how they work), it is only about tooling and code completion for Typescript users

@thetutlage Hi! Should I make pull requests incrementally after each step completed, or should I make many pull requests at once when everything is ready?
For now, lucid types are ready at that point when you can use migrations with .ts extension, enjoying full code completion, including chained methods. See https://github.com/anurbol/adonis-lucid/commits/ts
What  I plan next is:
add TS support for controllers and models
add docs about enabling and using TS
(quite complex one) add option to generate migrations, controllers, models with TS extension (and corresponding minor syntax differences). I though about this, how to do it the best way: easy for TS users and hidden from JS users, I think TS users should have command e.g. adonis ts:init, which creates config/typescript.ts. That file have something like:
{
  generateTsFilesFor: {
    migrations: true,
    controllers: true,
    models: true,
  }
}

So when user runs e.g. adonis make:model, cli should parse that config and generate files with correct extension.
What do you think (question is at the beginning)?

@anurbol This looks awesome :)
I believe, we can do it incrementally inside a new branch, so it will be easier to track the changes and also simple for you to work incrementally on it.
I have created a typescript branch in Lucid and framework repo. Feel free to create a PR anytime

Thanks for the response! Great!!

@anurbol https://github.com/tiansin/adonis-quick-start-typescript
An interim solution, the application code using the typescript to write the code, then use auxiliary definition file to realize automatic.
I put all the typescript code in the src directory, the directory structure should be consistent with the application file structure, so the compiled files can link up.

@tiansin Its a shame that I failed to think of this solution. Great one


The get user interface returned data that does not conform to the swagger specification.

As shown in the figure, get-user the interface returns data in html format instead of standard json data, which is not Swagger-compliant and does not facilitate api usage it should look like this:

@zuohuadong it's because you lacked name parameter besides owner, which triggered the backend panic. Beego crashes will render HTML. This is why you see HTML instead of JSON. Removing backend panic usage is in the plan, but not very soon to be done

@hsluoyz Even if the argument is missing, it should return a JSON error instead of HTML, which will cause the downstream to not catch the error. It's not a good habit

@zuohuadong can you give an URL from: https://github.com/beego/beego/tree/v1.12.12 ?
https://pkg.go.dev/github.com/dlt/beego/middleware is not official link


Attachment size is not right for attachment is eml or outlook item.

when I use the nice library to get eml attachment  I found there are missed attachment when both attachment is eml or outlook msg. Could you pls check that?
here is my sample email eml file
test_for_attachment.zip
I try to get the eml attachement should be two but only get one.

by reading your code seems the attachment used a map for store the attachmentList. if name is both "unnamed" will missed the attachment.  Can you help to fix the issue? thanks

Which version are you on? I'm simply getting an error, so I'm not even able to read the eml file (not good either).

Duplicate of #351

This all should work fine now in 6.7.0, which was just released. Please give it a try.


Scroll on real time? [enhancement]

Currently using 7.0.6
I am using didScrollToDateSegmentWith to change the header of my calendar but if I keep scrolling, the header function didScrollToDateSegmentWith won't be called until I stop scrolling. I would rather have the option to call this function every time that I scroll to a new month. So the header would update right away with every month I go by and not only with the last month.

hmm. Ok. in that case try implementing this delegate funciton ->
func scrollDidEndDecelerating(for calendar: JTAppleCalendarView) {
   let visibleDates = calendarView.visibleDates()
   // etc etc
}

This doesn't cut it if you have multiple months where you can keep scrolling but still want to see what month you are in without stopping. Maybe I will send a PR with this feature...

OK sorry. maybe i misunderstood the problem. I thought that delegate function will do it.
Let me double check this now.

I now see what you mean. I not sure i'll be able to get to that upgrade right away. But i'll take a look when i'm free.

OK hey, here is an idea, let me know if this can work for you.
in the cellForItemAt function, cells are configured.
Inside that function you have a parameter called cellState.
Is it possible you can try this?
class MyViewController: UIViewController {
   ...
   ...
   var currentMonthIndex: Int ?
func calendar(_ calendar: JTAppleCalendarView, cellForItemAt date: Date, cellState: CellState, indexPath: IndexPath) -> JTAppleCell {
   let dataSection = cellState.dateSection()
   if dataSection.month != currentMonthIndex {
      currentMonthIndex = dataSection.month
      let visibleDates = calendarView.visibleDates()
      // etc etc
   }
   ...
   ...
}

I just tested it.
I think it should be able to work for you.
Let me know.

closing issue now.
Let me know. And if unsolved, i'll re-open it

I get limited control with this... I want to change my outside header right when the top most visible month header reaches the bottom of my outside header... this is a problem since I can not access the scrollViewDidScroll using the contentOffset...

@JesusAdolfo can you try master branch?
we recently added the scrollViewDidScroll function.
func calendarDidScroll(_ calendar: JTAppleCalendarView)

put this is podfile to try the master branch ->
pod 'JTAppleCalendar', :git => 'https://github.com/patchthecode/JTAppleCalendar.git', :branch => 'master'

Alright this is is awesome... Haven't tried yet but definitely will once you release as a new version. I extended my own delegate from the existing one to get the contentOffset 👍
protocol CustomCalendarDelegate: JTAppleCalendarViewDelegate
When are you releasing a new version with this included?
It would make my code cleaner, no need to extend your protocol

Well, first i have to test it, but you can use that branch till its released. My master branches as stable enough.


[BUG] RecordPageHandler initialize failed after observer restart.

Create a table and insert some rows, then stop observer.
➜  build ✗ ./bin/obclient -s miniob.sock
miniob > create table just_a_test (id int, id2 int);
SUCCESS
miniob > insert into just_a_test values (1, 1);
SUCCESS
miniob > insert into just_a_test values (2,2);
SUCCESS
miniob > exit
---
### then stop observer
➜  build ✗ ./bin/obclient -s miniob.sock
miniob > select * from just_a_test;
id | id2

Restart observer, then select from the table created before. It return no record.
RecordPageHandler will be nullptr and record_page_iterator_ is invalid.
RC RecordFileScanner::fetch_next_record()
{
  RC rc = RC::SUCCESS;
  if (record_page_iterator_.is_valid()) {
    // 当前页面还是有效的，尝试看一下是否有有效记录
    rc = fetch_next_record_in_page();
  .......
}

How did you startup the observer?

./observer -s ../miniob.sock -f ../../etc/observer.ini

Currently, MiniOB doesn't support durability and if you stop the observer, the data maybe lost. But this issue will be fixed in the future.
You can use sync command to dump the data in the memory into disk now.


Add type annotations to _registry.py

Description
Add type annotations to _registry.py so that it will pass mypy --strict.
Comment
I was reading the code and felt that this module would be easier to understand with type annotations. Therefore, I went ahead and added the annotations.
The idea with this PR is to start small to see if we can align on how to annotate types. I've seen people in the past disagree on how strictly to annotate the code base, so before spending too much time on this, I wanted to check if you agree @rwightman.
Most of the added types should be straightforward. Some notes on the non-trivial changes:

I made no assumption about the fn passed to register_model, but maybe the type could be stricter. Are all models nn.Modules?
If I'm not mistaken, the type hint for get_arch_name was incorrect
I had to add a # type: ignore to mod.__all__ = ...
I made some minor code changes to list_models to facilitate the typing. I think the changes should not affect the logic of the function.
I removed list from list(sorted(...)) because sorted returns always a list.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@BenjaminBossan tx, the registration fn is regestering functions that return models (which are always nn.Module, yes).
I've actually debated whether to add a timm intermediate model class between nn.Module and all of the module instances that covers the additional timm specific API (make it clearer what's expected for model contribs)....

I've actually debated whether to add a timm intermediate model class between nn.Module and all of the module instances that covers the additional timm specific API (make it clearer what's expected for model contribs)....

Interesting, is this specific API documented somewhere? When it comes to typing, it may be possible use a Protocol. That way mypy* can check if the registered models conform to the protocol but users don't need to inherit from any extra class.
*I think there were some issues with decorators in the past, so I'm not completely sure if it works now


Add recommendation about Taas

Add recommendation about Taas

Are you in any way affiliated with this service?

@KnorpelSenf I am the creator of this service

We can't add links to third-party closed source services. Examples section is intended to contain examples of TDLib usage in different languages. A link can be added there if it points to source code of your service or even a part of the source code, which interacts with TDLib.


Fixed log loops and signal errors

(fixes https://github.com/Inria-Asclepios/music/issues/382, replaces https://github.com/Inria-Asclepios/medInria-public/pull/162)
This PR fixes some of the issues we've had with log files:

The log messages are sent by an asynchronous signal using char*, which leads to debug messages with strange characters because the data containing the message was trashed in memory. To fix this I had used a previous solution (https://github.com/Inria-Asclepios/medInria-public/pull/162) where I made the signal synchronous, which guaranteed that the message was being written before the data was trashed. This worked but introduced another bug: if the logger sends its own message during the process, it caused a recursion into the logging process which provoked a freeze. I dropped this idea and instead used QThreadLocalStorage to track if a same thread enters the logging process recursively. If that happens, the second message is not written (so any messages sent by the logger itself will be dropped).
Since I removed the correction that prevented the message from being trashed in memory, I added another fix for that: instead of using char*, the signal uses QString&, and for this type of data Qt automatically does a copy to ensure integrity of the data being sent.

So now there should be no freeze, and the logging is kept in asynchronous mode, which prevents the processes from lagging because they don't have to wait for the messages to be written (as before).

Ok for me (and i needed this PR in order to test something else, it's working)

It seems to works.
Nice job.


Window drag'n'drop (move) not working anymore

It stopped working after migration to v5.1
I've found that Chromely.NativeHost.DwmFramelessController contains bug in MouseLLHook_MouseMoveHandler code.
SetWindowPos(_windowHandle, IntPtr.Zero, 0, 0, eventArgs.DeltaChangeSize.Width, eventArgs.DeltaChangeSize.Height,

x and y arguments always set to 0, so it moves window to top left border of screen. But eventArgs contains correct values. Changing this code like this will fix behavior to expected:
SetWindowPos(_windowHandle, IntPtr.Zero, eventArgs.DeltaChangeSize.Width, eventArgs.DeltaChangeSize.Height, 0, 0,

Also DeltaChangeSize is too contextual name for MouseMoveEventArgs.
I'd recommend to change its properties from
public Size DeltaChangeSize { get; set; }

to
public int DeltaX { get; set; }
public int DeltaY { get; set; }

Another problem is:
!eventArgs.DeltaChangeSize.IsEmpty```

that condition will not allow to place window at 0:0 position.

@donotcodeit  good catch 👍
I did not really have a chance to test yet, so if you find something else with the change, will appreciate your feedback.
Thanks.

Thank you for quick fix!


Change apiTimeout type to Optional

To keep backward compatibility with updates/upgrades of existing env, this parameter can not be type Required.
Error msg on update env:
message: 'error validating existing CRs against new CRD''s schema for "openstackcontrolplanes.core.openstack.org":                       error validating core.openstack.org/v1beta1, Kind=OpenStackControlPlane "openstack/controlplane":
    updated validation is too restrictive: [].spec.octavia.template.apiTimeout: Required
    value'

/ok-to-test

/cherry-pick 18.0-fr1


Add a couple section headers in language overview doc

I added mention of the Zed language // comment syntax back in #4308 after a community user reported not realizing this concept existed (#4302). Another community user just pointed out via https://github.com/brimdata/zed/issues/4310#issuecomment-2214786576 that the way it's covered in the docs now doesn't currently bubble up nicely in the search results. I'm not 100% certain if their proposed change of adding a section header would improve that, but rather than investing time into setting up a test sandbox to investigate, I spent some time re-reading the page to see if adding a header makes sense regardless.
Looking at it through that lens, indeed, I can see that the coverage as it's been feels a little "tacked on", but I was always had trouble coming up with a better idea. Having a wholly separate page under "Language" at the same level as "Expressions", "Operators" etc. just to briefly mention comments feels like overkill. I also don't see a different existing page into which it would graft nicely as a header-worthy section, though I'm open to suggestion.
The "Overview" page should surely be brief, so I respect the urge to avoid lots of section headers there. However, I did ultimately find that adding a "Search and Analytics" header feels pretty natural to me, so once I've got that, the header for Comments feels a bit more natural, too.
As for the user's point in #4302  about if we should divulge the gotcha here about multi-line /* */ comment syntax being available in ZSON but not Zed language, I'm holding off on that for the moment in hopes it might be easier to just enhance the parser. 🤞

Now that this has been merged and the Zed docs site has been re-indexed by our search tool, I'm pleased to say that a search for "comment" is indeed now showing the top hit being the section in the Language Overview page. 🎉   Thanks for the idea @chrismo!

(Alas, at this moment that's only true if the "Next" docs revision is selected in the drop-down, since these changes weren't in the docs tagged for the most recent GA release. But I do hope to have a new release out quite soon, so this'll continue to improve.)

Looks great!


[AIRFLOW-1870] Enable flake8 tests

Flake8 tests now run for diffs
JIRA

[XXX] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1870

Description

[X] Here are some details about my PR, including screenshots of any UI changes:

flake8
Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@Fokko

Very nice @bolkedebruin, maybe also enable a nice landscaping tool. For example https://landscape.io/, https://codeclimate.com/ or https://www.codacy.com/? I don't have a lot of experience with Python tooling, but I'm willing to invest some time to make a comparison.

(imho out of scope of this PR): Landscape we tried, but was very flaky with uptime and reports are kind of unclear. Flake8 reports a lot of these errors as well. But anyways be my guest. Please note activating requires Apache INFRA support and site wide write access is not allowed.

Codecov Report

Merging #2829 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2829      +/-   ##
==========================================
- Coverage   73.84%   73.83%   -0.01%
==========================================
  Files         159      159
  Lines       12076    12076
==========================================
- Hits         8917     8916       -1
- Misses       3159     3160       +1

Impacted Files
Coverage Δ

airflow/models.py
87.31% <0%> (-0.05%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e7b0ab...518cd52. Read the comment docs.

Good point, LGTM. Thanks for adding Flake8 to the build again. Cheers