Using export_method as "development" for Appstore build.

Hello,
I was trying to export the archive(generating ipa) for the AppStore distribution, I have the AppStore profile , but the problem is that the Appstore profile is created using developer profile, so when I use export_method as "app-store", fastlane complains that "Distribution Profile not found" however I can successfully export using "development" in export_method.
My question is , will there be any problem if I use export_method as "development", instead of "app-store" for Appstore build, or is there a work around for this problem?
Thanks in advance.
Question Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Question Subject

Question Description

🚫 fastlane environment 🚫
Stack

Key
Value

OS
10.13.2

Ruby
2.4.1

Bundler?
false

Git
git version 2.10.1

Installation Source
~/.rvm/gems/ruby-2.4.1/bin/fastlane

Host
Mac OS X 10.13.2 (17C88)

Ruby Lib Dir
~/.rvm/rubies/ruby-2.4.1/lib

OpenSSL Version
OpenSSL 1.1.0h  27 Mar 2018

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
9.2

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL

LANGUAGE

fastlane files:
`./fastlane/Fastfile`
# Customise this file, documentation can be found here:
# https://github.com/fastlane/fastlane/tree/master/fastlane/docs
# All available actions: https://docs.fastlane.tools/actions
# can also be listed using the `fastlane actions` command

# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`

# If you want to automatically update fastlane if a new version is available:
# update_fastlane

# This is the minimum version number required.
# Update this, if you use features of a newer version
fastlane_version "2.26.1"

default_platform :ios

#ENV["DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS"] = "-t DAV"

platform :ios do

before_all do

#  sh("export http_proxy=proxyvipecc.nb.ford.com:83")
#   sh("export https_proxy=proxyvipecc.nb.ford.com:83")

end

  desc "Submit a new Beta Build to Apple TestFlight"
  desc "This will also make sure the profile is up to date"
  lane :beta do

gym(
       scheme: "TestFastlane",
       configuration: "Release",
       export_method: "development",
       skip_build_archive: false,
       output_directory: "./build",
       build_path: "./build",
       disable_xcpretty: true,
      #  xcargs: "PROVISIONING_PROFILE_SPECIFIER='Ford Pass Europe-DEV'"
      #  export_options: {
      #    provisioningProfiles: {
      #      "com.ford.fordpasseu" => "Ford Pass Europe-DEV"
      #      }
      #     }

)

build_number = latest_hockeyapp_version_number(
	    api_token: "2d3264e0eee94e1fbb31d3f5a5b1b4db",
	    app_name: "Upload Archive testing"
  )
  increment_build_number(
        build_number: build_number + 1,
        xcodeproj: "TestFastlane/TestFastlane.xcodeproj"
	)

# upload to HockeyApp
  hockey(
     api_token: "2d3264e0eee94e1fbb31d3f5a5b1b4db",
     public_identifier: "3368c958f60a457194946ef9314f6faf",
#     dsym: "../TestFastlane/build/TestFastlane.dsym.zip",
#     bundle_version: "1",
#     #bundle_short_version: "1",
#     notes_type: "1",
#     status: "2",
#     notify: "2",
#     #create_update: true,
#     upload_dsym_only: true
 )
    after_all do
        #Call the script to push to hockeyapp
        # sh("./hockey.sh")
    end

  end

end

# More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Platforms.md
# All available actions: https://docs.fastlane.tools/actions

# fastlane reports which actions are used
# No personal data is recorded. Learn more at https://github.com/fastlane/enhancer

`./fastlane/Appfile`
app_identifier "com.ford.TestingApp" # The bundle identifier of your app
apple_id<EMAIL_ADDRESS># Your Apple email address

team_id "[[DEV_PORTAL_TEAM_ID]]" # Developer Portal Team ID

# you can even provide different app identifiers, Apple IDs and team names per lane:
# More information: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Appfile.md

fastlane gems

Gem
Version
Update-Status

fastlane
2.59.0
🚫 Update available

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.1.0

executable-hooks
1.4.2

bundler-unload
1.0.2

rubygems-bundler
1.4.4

bundler
1.16.1

io-console
0.4.6

slack-notifier
1.5.1

CFPropertyList
2.3.5

claide
1.0.2

colored2
3.1.2

nanaimo
0.2.3

xcodeproj
1.5.2

rouge
2.0.7

xcpretty
0.2.8

terminal-notifier
1.8.0

unicode-display_width
1.3.0

terminal-table
1.8.0

plist
3.3.0

public_suffix
2.0.5

addressable
2.5.2

multipart-post
2.0.0

word_wrap
1.0.0

tty-screen
0.5.0

babosa
1.0.2

colored
1.2

highline
1.7.8

commander-fastlane
4.4.5

excon
0.59.0

faraday
0.13.1

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20170404

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

fastimage
2.1.0

gh_inspector
1.0.3

mini_magick
4.5.1

multi_json
1.12.2

multi_xml
0.6.0

rubyzip
1.2.1

security
0.1.3

xcpretty-travis-formatter
0.0.4

dotenv
2.2.1

faraday_middleware
0.12.2

uber
0.1.0

declarative
0.0.10

declarative-option
0.1.0

representable
3.0.4

retriable
3.1.1

mime-types-data
3.2016.0521

mime-types
3.1

little-plugger
1.1.4

logging
2.2.2

jwt
1.5.6

memoist
0.16.0

os
0.9.6

signet
0.7.3

googleauth
0.5.3

httpclient
2.8.3

google-api-client
0.13.6

json
2.1.0

openssl
2.0.3

generated on: 2018-05-16

@jelangba-ford Building for the app-store will require that the "provisioning profile" mapping get set (which based upon your Fastfile, it looks like you had commented out). You will either want to uncomment that out and make sure things are set correctly there or use the preferred method of calling match or sigh before calling gym. match and sigh will both create the proper proper provisioning profile mapping for you that should help you build and distribute your app properly 😊


if k8s internal ip is just a IPv6 address，cilium datapath is vxlan，how does cilium_vxlan work?

The srcNodeIP and dstNodeIP is IPv6 address which is got form k8s internal IP? am I right?

Correct


Add needed rerferences to KeyVault

Add needed references to KeyVault

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@mschray : Thanks for your contribution to the Azure documentation! The author, @mattchenderson, has been notified to review your proposed change.

@mschray Good catch - thanks! My only fear with this change is that it might imply Key Vault dependencies are required to use Microsoft.Azure.Services.AppAuthentication. What do you think about a parenthetical to the Key Vault reference that explains why it's needed here?
Regardless, let's merge this in. #sign-off


It just took a little while for the yellow dot to show up.


DCOS-14759: deprecate marathon Healthcheck protocols

Checklist

[x] Did you add a JIRA issue in a commit message or as part of the branch name?
[x] Did you add new unit tests?
[ ] Did you add new integration tests?
[ ] If this is a regression, did you write a test to catch this in the future?

This PR is deprecating Marathon healthcheck protocols http, https and tcp. And also ensures that unsupported health checks do not break the ui:

For testing purposes here is a JSON:
{
  "id": "/myapp",
  "cmd": "sleep 1000;",
  "instances": 1,
  "cpus": 1,
  "mem": 2048,
  "healthChecks": [
    {
      "path": "/status",
      "portIndex": 0,
      "protocol": "HTTP",
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false
    }
  ],
  "portDefinitions": [
    {
      "protocol": "tcp",
      "port": 0
    }
  ]
}

Closes DCOS-14759

@tobilg could you please check the wording?

@orlandohohmeier @Poltergeist I'd suggest to put "Deprecated Marathon health checks (read-only)" in the header.

This works nicely! Great work!

Made that one prettier!
@jfurrow please hit that Approve button if it still looks good to you!


User guidance or module for DNS

User Story
As a modernisation platform customer
I want an easy, consistent way to provision DNS names & certificates
So that I can use existing best practice, and not have to invest time in understanding this
Assumptions / Hypothesis
If we provide users with clear guidance including cut and paste snippets or a module
Then we will get less questions from customers, who will need to invest less time, who will get a consistent outcome
Because they will have an easier process to follow
Value
Our customers have a consistent need to provision domain names and SSL certificates.
We have seen a few differing approaches from customers, and that customers have had to invest engineering time to create these DNS records & certificates
Questions

Do we want code snippets or a reference-able module?
Do we want to provision certificates & domain names in one go?
How will we handle production certificates that are not generated by resources we administer?

Definition of done

[x] readme has been updated
[x] user docs have been updated
[x] another team member has reviewed
[x] tests are green

Reference
How to write good user stories
DNS in Modernisation Platform

https://github.com/ministryofjustice/modernisation-platform/pull/1675
Basic guidance for the most common use cases here, we may want to put this into a module or expand the guidance further for edge cases

New issue for creating a module here - https://github.com/ministryofjustice/modernisation-platform/issues/1679


Secor to S3 as parquet error.

I try to write Kafka massages to S3 using the S3 messageParser:
Secor Version 0.26
We saw the error in production:
Code Java.
package com.pinterest.secor.parser;
import com.pinterest.secor.common.*;
import com.pinterest.secor.message.*;
import net.minidev.json.*;
import org.slf4j.*;
import java.text.*;
import java.util.*;
public class RsdParser extends MessageParser {
    private static final Logger LOG = LoggerFactory.getLogger(RsdParser.class);
    private static final String DATE_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'";
    private static final String PATH_START = "PATH_START";
    public RsdParser(SecorConfig config) {
        super(config);
    }
    public String[] extractPartitions(Message message) {
        try {
            JSONObject jsonObject = (JSONObject) JSONValue.parse(message.getPayload());
            jsonObject = (JSONObject) JSONValue.parse(jsonObject.get("message").toString().split(" $$$ ")[0]);
            String intermediatePath = ((JSONObject) jsonObject.get("secor")).get("intermediatePath").toString();
            if (intermediatePath.contains(" ")) throw new Exception("path has illegal characters: " + intermediatePath);
            return intermediatePath.split("/");
        }
        catch (Exception e) {
            LOG.warn("could not get partition from message: " + e.getMessage() + ". " + message.getPayload());
            return new String[]{PATH_START, new SimpleDateFormat(DATE_FORMAT).format(new Date())};
        }
    }
}

eror code:
java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:99) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:132) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:132) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:132) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:131) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:131) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:131) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:130) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
he's appear few time but create crash of Spark jobs.
Thanks

Parquet conversion does needs lots of memory, did you try to increase the
JVM memory size or try to cut down the number of columns before conversion?
On Thu, Sep 26, 2019 at 12:33 AM Yehoshoua Dahan<EMAIL_ADDRESS>wrote:

I try to write Kafka massages to S3 using the S3 messageParser:
Secor Version 0.26
We saw the error in production:
Code Java.
package com.pinterest.secor.parser;
import com.pinterest.secor.common.;
import com.pinterest.secor.message.;
import net.minidev.json.;
import org.slf4j.;
import java.text.;
import java.util.;
public class RsdParser extends MessageParser {
private static final Logger LOG = LoggerFactory.getLogger(RsdParser.class);
private static final String DATE_FORMAT = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'";
private static final String PATH_START = "PATH_START";
public RsdParser(SecorConfig config) {
super(config);
}
public String[] extractPartitions(Message message) {
try {
JSONObject jsonObject = (JSONObject) JSONValue.parse(message.getPayload());
jsonObject = (JSONObject) JSONValue.parse(jsonObject.get("message").toString().split(" $$$ ")[0]);
String intermediatePath = ((JSONObject) jsonObject.get("secor")).get("intermediatePath").toString();
if (intermediatePath.contains(" ")) throw new Exception("path has illegal characters: " + intermediatePath);
return intermediatePath.split("/");
}
catch (Exception e) {
LOG.warn("could not get partition from message: " + e.getMessage() + ". " + message.getPayload());
return new String[]{PATH_START, new SimpleDateFormat(DATE_FORMAT).format(new Date())};
}
}
}
eror code:
java.lang.OutOfMemoryError: Java heap space at
java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) at
java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120) at
kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:99) at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:132)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:132)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:131)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:131)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:131)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at
kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:130) at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
at
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
he's appear few time but create crash of Spark jobs.
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964?email_source=notifications&email_token=ABYJP76AJFNECTZRSWMHELDQLRQTFA5CNFSM4I2WGOT2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNZXFNQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABYJP77URZPXHYOTOQXLGZTQLRQTFANCNFSM4I2WGOTQ
.

Here are the CPU and memory graphs - the crashes happen very regularly, every few days:
Memory - looks like leaking:

CPU

Actually it looks like "Secor is leaking memory through DeleteOnExit" #785 issue.

any updates ?

This is interesting.  If you can reproduce the problem, can you turn on the debug logging category for FileRegistry.java which will prints out all the files added for deleteOnExit.
It would be interesting to see why there will be so many files added, we have never seen the memory problem on DeleteOnExitHook before and DeleteOnExitHook doesn't have a method to clear the map either.  One suspicion I have is the file size is set to too small (secor.max.file.size.bytes and secor.max.file.age.seconds) so it ends up creating too many files.

Henry,
thank you for answering!
The problem is happening very regularly on our prod env,
see here - 6 times on 3 nodes over the last week:

We process about 500K files daily among all 3 nodes, each file about 100K in size - does it qualify as small?
I'll try to enable debugging and keep you posted.
It might take some time since this is production.

How much memory do you configure for the java process?  From https://github.com/pinterest/secor/issues/785, looks like DeleteOnExitHook took 260MB which is not that big.   If you create 500K files and assume each file path takes 100 bytes, you will register 50MB each day with DeleteOnExitHook

Wow... this is an embarrassment :)
We don't set the max heap size....
Thank you for pointing it out!
Se we are running on the defaults:
java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
intx CompilerThreadStackSize                   = 0                                   {pd product}
uintx ErgoHeapSizeLimit                         = 0                                   {product}
uintx HeapSizePerGCThread                       = 87241520                            {product}
uintx InitialHeapSize                          := 132120576                           {product}
uintx LargePageHeapSizeThreshold                = 134217728                           {product}
uintx MaxHeapSize                              :=<PHONE_NUMBER>                          {product}
intx ThreadStackSize                           = 1024                                {pd product}
intx VMThreadStackSize                         = 1024                                {pd product}
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
We run secor on 8G instances, so I can set max heap to at least 4G

I will post in a a few days if this improved the situation (e.g. if we had crashes)
Thank you again for the advice!

Great news - a week without a single crash, on all three nodes.
The memory usage however grows steadily. So far I'm afraid that we just pushed away how frequently the crash happens. I will update the ticket in a few weeks once it is clear.

Will you be able to do some heap dump or just object counts (which is less intrusive) on your system?  I think it's more likely the parquet side's memory consumption is far more than secor's

Henry, thank you - how do I count objects?
I want to have some non-intrusive metric that I can run periodically.
Currently (after maxmem fix) we have crashes not so often, but each instance still crashes between 6 and 15 days.

Try jmap: https://docs.oracle.com/javase/7/docs/technotes/tools/share/jmap.html, but not actually use -dump to generate the heap dump which will takes too much time.  Use '-histo' to generate the histogram of the heap which will include number of objects for each class, this is much faster.

Thank you Henry!
here is the /opt/jdk/bin/jmap -histo 20 for the 13-day java secor process (soon-to-die)
secor_jmap_histo_13days.log
and this is just a 2-day process (for comparison)
secor_jmap_histo_2days.log
Please let me know if some particular count is interesting to you - I can count it over time on the same process.

There are lots of character arrays which might come from String (6M char
arrays taking 2.8GB averaging 475 bytes each array), but it's not
conclusive to say it's from delteOnExit()'s hash map of file names.  For
DeleteOnExit, the map storage is the HashMap (the map variable in
LinkedHashSet), but I only saw 46K HashMap$Node, it's way less than 6M char
arrays
num     #instances         #bytes  class name

1:       6007380<PHONE_NUMBER>  [C
2:       5387565      215502600  java.util.LinkedHashMap$Entry
3:       1041602      161973168  [B
4:       5822839      139748136  java.lang.String
5:        140865      132649984  [I
6:         46626       37827656  [Ljava.util.HashMap$Node;
On Thu, Jul 9, 2020 at 1:41 PM David<EMAIL_ADDRESS>wrote:

Thank you Henry!
here is the /opt/jdk/bin/jmap -histo 20 for the 13-day java secor process
(soon-to-die)
secor_jmap_histo_13days.log
https://github.com/pinterest/secor/files/4899402/secor_jmap_histo_13days.log
and this is just a 2-day process (for comparison)
secor_jmap_histo_2days.log
https://github.com/pinterest/secor/files/4899407/secor_jmap_histo_2days.log
Please let me know if some particular count is interesting to you - I can
count it over time on the same process.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964#issuecomment-656340579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP76SMEZLFEXLD2UAMADR2YTN5ANCNFSM4I2WGOTQ
.

The other observation: the number of char arrays (6M) is very close to the
number of LinkedHashMap$Entry (5.3M), it looks like the memory consumption
is from a LinkedHashMap of String (either String is the key or the value),
but File.deleteOnExit() is using HashMap, not LinkedHashMap
On Fri, Jul 10, 2020 at 4:51 PM Henry Cai<EMAIL_ADDRESS>wrote:

There are lots of character arrays which might come from String (6M char
arrays taking 2.8GB averaging 475 bytes each array), but it's not
conclusive to say it's from delteOnExit()'s hash map of file names.  For
DeleteOnExit, the map storage is the HashMap (the map variable in
LinkedHashSet), but I only saw 46K HashMap$Node, it's way less than 6M char
arrays
num     #instances         #bytes  class name

1:       6007380<PHONE_NUMBER>  [C
2:       5387565      215502600  java.util.LinkedHashMap$Entry
3:       1041602      161973168  [B
4:       5822839      139748136  java.lang.String
5:        140865      132649984  [I
6:         46626       37827656  [Ljava.util.HashMap$Node;
On Thu, Jul 9, 2020 at 1:41 PM David<EMAIL_ADDRESS>wrote:

Thank you Henry!
here is the /opt/jdk/bin/jmap -histo 20 for the 13-day java secor process
(soon-to-die)
secor_jmap_histo_13days.log
https://github.com/pinterest/secor/files/4899402/secor_jmap_histo_13days.log
and this is just a 2-day process (for comparison)
secor_jmap_histo_2days.log
https://github.com/pinterest/secor/files/4899407/secor_jmap_histo_2days.log
Please let me know if some particular count is interesting to you - I can
count it over time on the same process.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964#issuecomment-656340579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP76SMEZLFEXLD2UAMADR2YTN5ANCNFSM4I2WGOTQ
.

Correction: I traced the File.deleteOnExit() once more, looks like the map
variable inside HashSet.java was initialized with LinkedHashMap (not the
regular HashMap), then it makes sense now.  The 5.38M LinkedHashMap$Entry
entries is probably related to DeleteOnExit(), we have accumulated 5.38M
file names.
On Fri, Jul 10, 2020 at 4:54 PM Henry Cai<EMAIL_ADDRESS>wrote:

The other observation: the number of char arrays (6M) is very close to the
number of LinkedHashMap$Entry (5.3M), it looks like the memory consumption
is from a LinkedHashMap of String (either String is the key or the value),
but File.deleteOnExit() is using HashMap, not LinkedHashMap
On Fri, Jul 10, 2020 at 4:51 PM Henry Cai<EMAIL_ADDRESS>wrote:

There are lots of character arrays which might come from String (6M char
arrays taking 2.8GB averaging 475 bytes each array), but it's not
conclusive to say it's from delteOnExit()'s hash map of file names.  For
DeleteOnExit, the map storage is the HashMap (the map variable in
LinkedHashSet), but I only saw 46K HashMap$Node, it's way less than 6M char
arrays
num     #instances         #bytes  class name

1:       6007380<PHONE_NUMBER>  [C
2:       5387565      215502600  java.util.LinkedHashMap$Entry
3:       1041602      161973168  [B
4:       5822839      139748136  java.lang.String
5:        140865      132649984  [I
6:         46626       37827656  [Ljava.util.HashMap$Node;
On Thu, Jul 9, 2020 at 1:41 PM David<EMAIL_ADDRESS>wrote:

Thank you Henry!
here is the /opt/jdk/bin/jmap -histo 20 for the 13-day java secor
process (soon-to-die)
secor_jmap_histo_13days.log
https://github.com/pinterest/secor/files/4899402/secor_jmap_histo_13days.log
and this is just a 2-day process (for comparison)
secor_jmap_histo_2days.log
https://github.com/pinterest/secor/files/4899407/secor_jmap_histo_2days.log
Please let me know if some particular count is interesting to you - I
can count it over time on the same process.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964#issuecomment-656340579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP76SMEZLFEXLD2UAMADR2YTN5ANCNFSM4I2WGOTQ
.

Maybe this PR: https://github.com/pinterest/secor/pull/1449 can fix the
problem.
On Fri, Jul 10, 2020 at 4:59 PM Henry Cai<EMAIL_ADDRESS>wrote:

Correction: I traced the File.deleteOnExit() once more, looks like the map
variable inside HashSet.java was initialized with LinkedHashMap (not the
regular HashMap), then it makes sense now.  The 5.38M LinkedHashMap$Entry
entries is probably related to DeleteOnExit(), we have accumulated 5.38M
file names.
On Fri, Jul 10, 2020 at 4:54 PM Henry Cai<EMAIL_ADDRESS>wrote:

The other observation: the number of char arrays (6M) is very close to
the number of LinkedHashMap$Entry (5.3M), it looks like the memory
consumption is from a LinkedHashMap of String (either String is the key or
the value), but File.deleteOnExit() is using HashMap, not LinkedHashMap
On Fri, Jul 10, 2020 at 4:51 PM Henry Cai<EMAIL_ADDRESS>wrote:

There are lots of character arrays which might come from String (6M char
arrays taking 2.8GB averaging 475 bytes each array), but it's not
conclusive to say it's from delteOnExit()'s hash map of file names.  For
DeleteOnExit, the map storage is the HashMap (the map variable in
LinkedHashSet), but I only saw 46K HashMap$Node, it's way less than 6M char
arrays
num     #instances         #bytes  class name

1:       6007380<PHONE_NUMBER>  [C
2:       5387565      215502600  java.util.LinkedHashMap$Entry
3:       1041602      161973168  [B
4:       5822839      139748136  java.lang.String
5:        140865      132649984  [I
6:         46626       37827656  [Ljava.util.HashMap$Node;
On Thu, Jul 9, 2020 at 1:41 PM David<EMAIL_ADDRESS>wrote:

Thank you Henry!
here is the /opt/jdk/bin/jmap -histo 20 for the 13-day java secor
process (soon-to-die)
secor_jmap_histo_13days.log
https://github.com/pinterest/secor/files/4899402/secor_jmap_histo_13days.log
and this is just a 2-day process (for comparison)
secor_jmap_histo_2days.log
https://github.com/pinterest/secor/files/4899407/secor_jmap_histo_2days.log
Please let me know if some particular count is interesting to you - I
can count it over time on the same process.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964#issuecomment-656340579,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP76SMEZLFEXLD2UAMADR2YTN5ANCNFSM4I2WGOTQ
.

@dovka Please try the PR: #1449

Thank you Henry!
Our third party usage policy requires us to use official releases.
For which release do you guys plan to pack PR: #1449?
When you think it will be ready?

David, I just cut a new release v0.29, it might take a day or two to show
up in maven central.   You can use that release
On Thu, Jul 16, 2020 at 1:08 PM David<EMAIL_ADDRESS>wrote:

Thank you Henry!
Our third party usage policy requires us to use official releases.
For which release do you guys plan to pack PR: #1449
https://github.com/pinterest/secor/pull/1449?
When you think it will be ready?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/secor/issues/964#issuecomment-659643420,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYJP77CCXIMQHON7JP5MN3R35M25ANCNFSM4I2WGOTQ
.

Henry - thank you so much!
I will push the release to prod and will let you know how it worked.
(should have a picture in couple of weeks or so)


Release version for Python 3

Please, release a version for Python 3, also on https://pypi.python.org/pypi/googlesearch/ Thanks

See also https://bugs.launchpad.net/ubuntu/+bug/1568509 and notify there once version for Python 3 is available.

really not much to do. Change except statements, very few (1?) print statement, use urllib.urlparse.urlencode(). About it.

The Google API that googlesearch used is no longer available, and barring a complete overhaul it is now defunct. Keeping this issue open to address it in case of a rewrite, but right now the package is withdrawn from PyPI.


Add support for IMDS credential extension

Adding support for server side credential validation during IMDS outage scenarios. This patch will defer validation of credential expiry to the service for cases where IMDS may not be capable of receiving new metadata updates.

Codecov Report

Merging #2664 (aa11b4e) into develop (b7a6f50) will increase coverage by 0.00%.
The diff coverage is 96.15%.

@@           Coverage Diff            @@
##           develop    #2664   +/-   ##
========================================
  Coverage    95.00%   95.00%
========================================
  Files           60       60
  Lines        12237    12258   +21
========================================
+ Hits         11626    11646   +20
- Misses         611      612    +1

Impacted Files
Coverage Δ

botocore/utils.py
93.66% <95.45%> (+0.01%)
:arrow_up:

botocore/credentials.py
98.65% <100.00%> (+<0.01%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7a6f50...aa11b4e. Read the comment docs.


Question: How to get Chromium Browsing data to ArchvieBox when it is in a VM?

I'm running Chromium on my desktop, completely separate from my ArchvieBox VM.
How would I get my browser history in to it?
The documentation assumes they're both running together?

They don't have to be running together, you can mount your CHROME_DATA_DIR inside the VM and pass it to archivebox.

They don't have to be running together, you can copy/mount your CHROME_DATA_DIR inside the VM and pass it to archivebox. Or export your history using https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/export_browser_history.sh on the desktop and pass the list of URLs to archivebox as text.

I can't run .sh on Windows, I need Linux for that.
The VM I refer to... is on another host system, I can't mount my chrome directory from my PC, over the network to another host....
I might be able to install Chrome on a network share... but that's getting messy here....?

I can't run .sh on Windows, I need Linux for that.

The new WSL2 should be able to run it. If you're not familiar you can install it here: https://docs.microsoft.com/en-us/windows/wsl/install-win10

The new WSL2 should be able to run it. If you're not familiar you can install it here: https://docs.microsoft.com/en-us/windows/wsl/install-win10

Lucky I have Windows 10 Pro. The average home user doesn't have this, I believe.
How am I going to run a Linux (say Ubuntu), mount Chrome data, and send my data to the remote VM (over LAN) (ArchiveBox) (I will not run ArchiveBox on my WSL2, on my desktop).
If this is so tricky, then I think the project needs some sort of API, or chrome extension or native Windows app.

ArchiveBox doesn't require WSL2, just the browser history export helper script. I don't use Windows, nor do I have a Windows machine to test on, you'll have to figure out a different way to get your history into ArchiveBox if you don't have a way to run the bash script or share your data dir with archivebox.

ArchiveBox doesn't require WSL2, just the browser history export helper script. I don't use Windows, nor do I have a Windows machine to test on, you'll have to figure out a different way to get your history into ArchiveBox if you don't have a way to run the bash script or share your data dir with archivebox.

Unlikely, even when I install Chrome, it doesn't let me specify an install location.
Only thing I can think of is move the Appdata app location to a shared NFTS drive... not ideal.
The project should really think of running this like an actual web service. If I have lots of data to "archive" I'm hardly going to run this on a desktop computer, unless I have tons of hdd's/USB harddrives. Average user doesn't.
Instead I have a dedicated server with tons of drives, and isolated VM environments, and I can't even send an API call or run a little script to "post" my chrome data to it....

I run it as a web service in a docker container on Linux, and on my desktop Mac. I don't archive all my history (it would quickly fill many terabytes). Much easier to archive a subset of your history using bookmarks or a tool like pocket / pinboard, and send that to your vm.

Much easier to archive a subset of your history using bookmarks or a tool like pocket / pinboard, and send that to your vm.

I have TB's of storage... not a problem.
I just can't get my browsing data to a remote machine for ArchiveBox.
The project in it's current state isn't fit for my purpose, which is sad for me. But I'm sure it works for many who only have a handful of sites to archive and don't have dedicated hardware.

No it's a dedicated VM running in the cloud, not on a local PC. Why not rsync your data dir to the server periodically?

No it's a dedicated VM running in the cloud, not on a local PC. Why not rsync your chrome data dir to the server periodically and then run the export script on the VM?

What folder exactly needs to go to the remote?
I am using a Chromium browser (not official Chrome). Userdata folder?

Yup, whatever folder contains the Default folder (your default chrome profile).
C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default  <- not this
C:\Users\<username>\AppData\Local\Google\Chrome\User Data  <- this one
Chromium is what archivebox uses (not chrome), so it should be fine, just make sure  you're on or near the latest version.

Closing for now.
I will need to check that Rsync works with Windows, perhaps I can write a batch script that'll send it to my NFTS share.
Thanks.

I tried cmd scp to copy files over to my remote linux VM, didn't work. I can't even zip my Chomium data as I get permission/read errors with it open (I'm not going to close it everytime it tries to copy /User Data/
I hope this project can be more friendly towards Windows.


Support for sourcemaps in aws-rum-web

Hello,
Are there any plans to support sourcemaps in CloudWatch RUM? In short I'd like to be able to understand the root cause of errors similar to this:

I'm looking for something similar to https://docs.datadoghq.com/real_user_monitoring/guide/upload-javascript-source-maps/

We (the CloudWatch RUM team) are exploring this, although it is outside the scope of the CloudWatch RUM web client.

Hello, Is their any progress on this ticket ?
Is it still being considered ?
Thanks

@qhanam is there any progress on this ticket?
This is potentially a deal breaker, forcing teams to use datadog, sentry or other solutions

It would be very cool to have this feature! In Sentry we can upload source maps via API, in NewRelic it's possible to upload them in UI. Generally, no matter how to upload, both variants are good.

+1 to this request, we need it to debug issues getting reported in RUM. e is unedfined is no where close to being useful here.


TP-271: Implement interdependent and linked models importer

The current importer, whilst very effective at extracting data from XML
in a simple to use manner, is naive to a some of the complexities
around the TARIC dataset and the changes that have been made to the
data models. Two prime examples of this are models which have been merged
and how foreign keys are transmitted.
Merged models has the consequence of the required data for each models being
split across several records within the incoming data. Whereas for Foreign Keys
have no guarantee of being processed in the correct order.
As a result there is no guarantee that at any one time the data given from a
TARIC record will be complete or ready for processing.
To account for this a pipeline has been constructed which consists of three new
concepts introduced and the parser concept partially redefined:

Parsers

Parsers have now been relegated to only handle extracting data from the XML
and packaging it for processing elsewhere. This keeps the parser classes
lightweight and simple to design.

Handlers

Handlers are at the other end of the pipeline from parsers. They take the parsed
data and match it with other data until a complete dataset for an object has been
obtained. They also handle fetching foreign keys. The end result of a handler should
be a database object. This is the most complex new concept introduced and comes with
a lot of customisability for specific models, however at it's simplest it should be
trivial to define.

The Nursery

The nursery is a caretaker process which effectively orchestrates the pipelines.
It receives data from parsers and matches them to the handlers. If the data turns
out to be incomplete (determined by how successful a handler is) it stores the data
to be matched against later inputs.

The Cache

The nursery needs to store the incomplete data for later processing, it does this in
a "cache". Currently this is implemented as a process level dictionary, but can easily
be adapted to some more persistent or distributed storage solutions down the road.
All the above works together to make an asynchronous data pipeline where partial data
for an object can be sent at any time and can be patched together at a later point.
Some example handlers and parsers have been defined to showcase this, specifically
the FootnoteAssociationGoodsNomenclatureHandler in the commodities handlers
showcasing how links work. And the FootnoteTypeHandler together with the
FootnoteTypeDescriptionHandler showcasing how interdependent models work.

Codecov Report

Merging #56 into master will decrease coverage by 0.69%.
The diff coverage is 55.17%.

@@            Coverage Diff             @@
##           master      #56      +/-   ##
==========================================
- Coverage   68.04%   67.34%   -0.70%
==========================================
  Files          93      100       +7
  Lines        3398     4002     +604
  Branches      218      245      +27
==========================================
+ Hits         2312     2695     +383
- Misses       1027     1239     +212
- Partials       59       68       +9

Impacted Files
Coverage Δ

footnotes/models.py
25.00% <ø> (-1.59%)
:arrow_down:

commodities/import_handlers.py
6.89% <8.69%> (-5.77%)
:arrow_down:

importer/taric.py
41.53% <27.27%> (ø)

importer/utils.py
28.00% <28.00%> (ø)

importer/nursery.py
37.50% <37.50%> (ø)

importer/handlers.py
54.54% <55.23%> (-2.28%)
:arrow_down:

importer/parsers.py
56.97% <56.97%> (ø)

certificates/tests/test_importer.py
100.00% <100.00%> (ø)

common/tests/test_models.py
100.00% <100.00%> (ø)

footnotes/tests/test_importer.py
100.00% <100.00%> (ø)

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e0a07e...b3c763b. Read the comment docs.

Continuing problems:
Ignoring the "special case" that is the commodity code tree, there is also still an issue with optional but existing foreign keys.
It's kind of hard to decide whether a key is optional or not in some cases. For example GoodsNomenclatureOrigin. There are definitely cases there where the key should be null. But the majority of cases it is required. But the null requirement means we don't check for it. More importantly the key comes in an "interdependent" record - this is a table that's been squashed into a single field. We don't really have a mechanism for waiting for interdependent records which are actually foreign keys. It's kind of a confusing state to be in.
I reckon there will be a relatively obvious path around more complex checks for whether a link is optional or not and how to wait (e.g. making the optional value either a bool or a callable which returns a bool). I don't really want to confront this issue here because it's a bit more niche and I want to keep things moving (ironic considering how long this branch took to get up).
Possible changes
There is some meta programming in this implementation... specifically in handlers. Effectively all handlers need to be registered with the nursery, so to cater for this we do it on class definition via the BaseHandlers metaclass. This is actually also a reasonable route for handling the forward-referenced dependency issue. If a Handler defines another Handler as a dependency, then the metaclass can inspect these dependencies and auto-register the reverse dependency relationship. It would reduce boilerplater, but I'm already well aware the behaviour appears magical to anyone unaware of the metaclass. So I hesitate to introduce more magic, but honestly I think I'd prefer it.


fix: do not return object as translation

Previous to this fix, if passing in a partial key as a translation, it would return the child object instead of handling it as a missing translation and returning the key.
Previously:
Translation file: {"admin": {"list": "Admin List"}}
{{'admin' | translate}} displayed [Object object] because it was getting {"list": "Admin List"}
After Fix:
Translation file: {"admin": {"list": "Admin List"}}
{{'admin' | translate}} will display 'admin' or go through the custom missing translation handler.

Thanks for your effort. But imo this is desired behaviour. If we render [object Object], you can easily detect that you are doing something wrong. For instance, what if we have the following translation file
{
    "Admin": "Admin"
}

And in the view we have
<h2>{{ 'Admin' | translate }}</h2>

So far so good! But if we refactor our translation file to something like this
{
    "Admin": {
        "Admin": "Admin",
        "Role": "Role"
    }
}

The current version will display [object Object] because we forgot to refactor our view. Your version on the other hand will display Admin.
So unless @ocombe thinks differently, I only see downsides to this approach.

Yeah I'm not sure about this either. You're supposed to be the master of your translations, you should know that some key contains an object (and use key.childKey).
Also some people might want to get an object (with get or instant) in their service and do stuff with it, it could be the intended behavior...
Like @SamVerschueren said, your change limits those possibilities and introduces a breaking change. An object is not really a missing translation. I understand that what you really want is to be able to go through the missing translations handler (if someone gives you bad translations), but fixing those errors at runtime are probably not the way to go, you should fix them in dev before you ship your code.

Oh, I totally forgot about the use case that you sometimes want to get an object back in your JavaScript with multiple translations so you can work with that one instead of getting values one by one.
Because of all these downsides, I'm going to close this PR. Feel free to provide other arguments why we should reconsider.

@ocombe @SamVerschueren,
No problem if you don't want to take the change, I just want to explain a bit more of the scenario I was in.
I had a generic facet component where sometimes the value for the facet was a known translation key, and sometimes it was user input which I wanted to display directly as is.
Without taking this change, you can never really have a spot which sometimes takes a translation key, and sometimes takes user input to be displayed directly, because there is no way to guarantee it won't clash with the beginning of a translation key.
If you guys are fine with that, I'm happy to work around my situation as I already have, by defining if the facet values should be translated or not.
Just another note, a user would still be able to get the raw translation object with translationService. getTranslation. I'm still not sure why you would ever want instant or the translation pipe to return anything but a string.
Thanks for taking the time to review and talk it out with me.


fix: include all hashlib.new inputs

This update is required when using tagreader package together with other packages that use hashlib.

The new input usedforsecurity was introduced in python 3.9


network issue between docker and virtualbox on mac osx

Expected behavior
Actual behavior
Information about the Issue
Steps to reproduce the behavior

...
...

i removed all, cleaned all files and reinstalled and I still get the same error.

I would recommend you use Docker for Mac instead of Toolbox:
https://docs.docker.com/docker-for-mac/install/

hi again,
I did as instructed to download the docker for Mac from this link:
https://docs.docker.com/docker-for-mac/install/
and then I created one container 'dev' successfully and then I tried another container 'default' and i get an error. Please see the snapshots attached


Question for Task 8

I have pulled the latest version and used autograder to test my code. For task 8, I get 4 points but still have the error message "cannot open resource". How can I fix the problem?

Could you paste your log here?

The error is caught and handled in util.py. As long as you get the points, you will be fine. If you want to fix the problem, just install corresponding font.

I also changed the error message to warning in the latest code

Thanks.


Can't build on latest Arch Linux

yay -Sy  frawk
.................
Compiling frawk v0.4.6 (/tmp/makepkg/frawk/src/frawk-0.4.6)
Compiling tikv-jemallocator v0.4.3
error[E0554]: #![feature] may not be used on the stable release channel
--> src/main.rs:2:43
|
2 | #![cfg_attr(feature = "unstable", feature(core_intrinsics))]
|                                                              ^^^^^^^^^
error[E0554]: #![feature] may not be used on the stable release channel
--> src/main.rs:3:43
|
3 | #![cfg_attr(feature = "unstable", feature(test))]
|                                                              ^^^^
error[E0554]: #![feature] may not be used on the stable release channel
--> src/main.rs:4:43
|
4 | #![cfg_attr(feature = "unstable", feature(write_all_vectored))]
|                                                              ^^^^^^^^^^^^^^^^^^
For more information about this error, try rustc --explain E0554.
error: could not compile frawk due to 3 previous errors
==> ERROR: A failure occurred in build().

Hi! Are you wondering how to fix this compilation error or to get the arch package to install?
I do not own the Arch package (maybe someone who does watches this repo and can weigh in) and don't have much experience in debugging install failures on Arch. For the former, I think the answer in that case is to either build on nightly or to build without the unstable feature.

Unfortunately I don't know rust/environment.
Just wanted to install and try it.


cors-anywhere and AWS Lambda

Hello! Has anyone succeeded in settings it up on AWS Lambda and can share some experiences? To me caching comes to mind, of course. However, to me an excellent low-cost use-case to to CORSify prepackaged or legacy APIs.

Hi Guys,
For reference, I am running this library on AWS using Elastic Beanstalk/Cloudfront/Route53. I use it to get images from HTTP image stores and and JSON data from APIs running CORS disabled webservers.
Note that I used Elastic Beanstalk over Lambda for two main reasons.

Technically at the time of writing, APIGateway/Lambda doesn't support image files (without base64 encoding into a JSON)
The high number of image requests by the application.

For lower usage applications that only need JSON or text based data, then I see no reason you couldn't deploy this using Lambda.
AWS Deployment tips w/ SSL
Some basic tips for deploying to AWS.
Elastic Beanstalk

Create a zip file with the lib folder, package.json and server.js.
Upload and Deploy zip file to Elastic Beanstalk
After deploy, go to Software Configuration and add an environment property for these as required:

CORSANYWHERE_BLACKLIST, CORSANYWHERE_WHITELIST, and/or CORSANYWHERE_RATELIMIT with your domain. E.g. www.example.com or subdomain
E.g. For CORSANYWHERE_RATELIMIT, in order to allow any subdomain of example.com to make any number of requests and block the rest: 0 1 /(.*\.)?example\.com/

Note the URL. E.g. myproxy.elasticbeanstalk.com

Cloudfront

Create a new Cloudfront distribution
Set the origin to the Elastic Beanstalk URL. E.g. myproxy.elasticbeanstalk.com
Add a free SSL certificate using AWS certificate store.
!! IMPORTANT !! Set the Query String Forwarding and Caching to Forward all
Add a CNAME entry. E.g. proxy.example.com

Route53

Add a CNAME record proxy.example.com that points to the Cloudfront URL

I've put this on a single instance t1.micro through Elastic Beanstalk and setup Cloudfront in front of it (without a custom cname), but I'm getting an error:
Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Kinda new to all of this. Can anyone suggest what I'm missing. I have a tool on HTTPS making a request to the cors anywhere server behind Cloudfront on HTTPS. Ultimately going to an HTTPS endpoint. I'm assuming I setup something wrong on Cloudfront.

Hi @DevanB
I like to think of these CORS issues in terms of the servers involved and the path the requests take.
Consider your client and three servers:
CLI) Your client (E.g. Chrome Browser)
CF) Cloudfront (CF)
CA) ELB running cors-anywhere
DS) Some end data server that has CORS disabled.
The "preflight request" is the OPTIONS request from your client asking a server if it is ok to send the actual request.
In normal operation the OPTIONS ("preflight request") path is:
CLI -> CF  (passthough) -> CA (respond OK) -> CF (OK) -> CLI (OK)
And the GET/POST/ETC path is:
CLI (URL) -> CF (URL) -> CA (URL add CORS headers) -> DS (GET DATA) -> CA (response data) -> CF (response) -> CLI (Parse Response)
The question is, which part is rejecting your OPTIONS request?
You need to check configuration.
CLI) Did you add the Access-Control-Allow-Origin header at the client?
CF) Did you allow CF to pass though OPTIONS? Is it enabled at all? Check the logs of C, is it passing the query to the cors-anywhere server?
CA) Did cors-anywhere get a request? Check the logs. Consider adding a debug statement (See line 277 /lib/cors-anywhere.js)
DS) Is the URL correct? Does it work querying directly? (Not part of a cross origin request)
Good luck!

Firstly,
I commend you for this kind of response. Detailed, thorough, helpful! Let me spit-ball here:
This issue arose after I put CF in place and used HTTPS. Prior to that, in certain uses, I had no issues.   So I'll try take this a layer at a time (and possibly ask questions as I progress):
CLI - I never had a "Access-Control-Allow-Origin" header in my header at the client. As I understand it, this is not something I add to my request, but rather to my response. This is what the OPTIONS call is "pre-flight checking", I think.
CF - I have checked and configured CF to use HTTPS only, and allow GET, HEAD, and OPTIONS methods. In addition, I have "Forward all, cache based on all" set for query string forwarding and caching. I am also using a Whitelist to cache selected request headers. In that whitelist I am have Origin whitelisted. I suppose I could add Access-Control-Allow-Origin, however, I still understand it to not be a request header, but rather a response header.
After checking my console again, I see I'm receiving a 504 via the OPTIONS call. And the error message points to that too by telling me that my response to preflight request doesn't pass access control. I know a 504 is a timeout error code. And looking at the log, the nginx logs have no entries with a timestamp remotely similar to the time I'm expecting. So perhaps, I have configured CF incorrectly. I have manually told it the origin domain name is the HTTP url to my beanstalk instance. Wondering if that is the issue.

@DevanB Tks.
CLI - Yes, sry, the Access-Control-Allow-Origin is returned by the server to the client to indicate the client is allowed to send the request.
CF - I've got two items in the whitelist... origin' and referer` to the whitelist. But note the referer is an old style of origin that I only use because I have a 3rd party JS library that still uses it.
Hmmm, not sure what else maybe check the attached screenshot of my configuration.

I am trying to set it up on an Ubuntu 18.04 AWS EC2 Instance with Node installed.
Running the server works, but i cannot make api calls over it. I always see an Forbidden:403 error

I am also interested in any experiences with running cors-anywhere on an AWS instance.  I can run it ok but I am not able to access the proxy.  I have opened the port the server is running but still I can't access from my local machine.  Is any other configuration required? Any help will be greatly appreciated.  I have searched many pages and can't find anything specific.  TIA

I have spent today getting cors-anywhere working on an EC2 instance, using Cloudfront for https, and Route53 for a friendly name. See https://github.com/ndg63276/smartathome/issues/2 for my notes. The key seemed to be getting my head around which bits were using http and which were using https, and then making sure those ports were correct, ie

Run cors-anywhere on port 80
open port 80 on the EC2
tell cloudfront to use HTTP on port 80 to talk to the origin
tell cloudfront to use HTTPS only for Viewer Protocol
Then set the cloudfront caching behaviour as mentioned in this thread (enabling OPTIONS, add "origin" to cache whitelist, Query String Forwarding and Caching set to "Forward All, Cache Based on All").

Hope that helps somebody.

I also tried to get it running on lambda. No success. Using AWS Apprunner was very easy.

I've extended this to work with aws_lambda and published it as a terraform module: lambda-cors-anywhere. It basically fetches this public repo & adds the lambda wrapper and deploys a lambda function to work with your customized allow origins.
See particularly this as the aws_lambda wrapper:
https://github.com/n-ae/terraform-aws-lambda-cors-anywhere/blob/main/aws_lambda_wrapper/index.js


Subscription to the fetchMappings operation is left open

@deeleman found this subscription is unnecessarily left open. This needs to be unsubscribed after complete to load mappings.
https://github.com/atlasmap/atlasmap-ui/blob/98ff5c16ae96aebdf8ceea0eb1a192483f250bdc/src/app/lib/atlasmap-data-mapper/services/mapping.management.service.ts#L108

Also don't forget to see if there're same issues elsewhere in the code and fix them.


Support event undo

See causality PR.
Basically what we do currently is insufficient, which is events to add = new events - old events. We need to also do events to remove = old events - current events from causality. The details are still being worked out, but this should be a priority after I get https://github.com/artsy/eigen/issues/1110 done.

not currently working as of June 6th QA

Is this happening in all cases or is there a specific way to reproduce this?

done for now. Individual updates will be tackled with new issues/epics


[2.7] Change scala bug workaround to work with Scala 2.13

Do not merge yet, this has to wait for the 2.7 release.
Deadbolt 2.7 will use Play 2.7. In Play 2.7 Scala 2.11 will be dropped, so there is no need anymore for that if - also because there will be Scala 2.13 support already so the fix has to apply to that major version as well.

Unfortunately Play 2.7 will not drop Scala 2.11, so I have to revert this change :disappointed:


anonymous access

Is it possible to get post information, from a public page such as this:
https://www.instagram.com/p/CT-cnxGhvvO/
without login?

Not with this lib, as this uses the android API which requires login.
You shared a weblink, and as long as it shows you the content without login screen you can access it but you'd need to build your own scraper or use another lib


feat(Data/Nat/Prime): add 2 theorems

We have 2 exists_dvd_of_not_prime, but not the other way round. I found these 2 theorems useful when proving a number is not a prime number.

Seems useful! Could you turn it into an iff lemma so that it can be used with rw?

Sure. I added 2 iff lemmas.

Rebased

whether the iff lemmas ought to be simp lemmas.

No.

I don't like the naming. Just adding a 2 is not very helpful. What about not_prime_of_dvd_of_le_of_lt, etc?

(Doesn't need new-feature or RFC labels for a standard PR adding theorems.)

I updated the name to be theorem not_prime_of_dvd_of_ne {m n : ℕ} (h1 : m ∣ n) (h2 : m ≠ 1) (h3 : m ≠ n) : ¬Prime n :=. compared to not_prime_of_dvd_of_ne_of_ne, the (h2 : m ≠ 1) is not in the name. When we think about it, or when we explain to others, we tend to omit the trivial m = 1 case. So, I also omitted in the name. I just feel it's more natural, but I'm totally OK if we want to stick to the name not_prime_of_dvd_of_ne_of_ne and similarly not_prime_of_dvd_of_le_of_lt for. (2<=m && m < n)

Let's merge this as is. Change names of existing material can be done in another PR.
bors merge


Remove Usage of Deprecated Ember.K Method

Please see https://github.com/emberjs/ember.js/issues/14746 for details.

Thanks @jherdman ! 👍


Add set moderator functions

Coverage remained the same at 62.303% when pulling 310f8502faba55eea94a85b5e48672f0ce7e0aa4 on cpacia:master into 5f728b384572e5dc7567aa6b748227b8acce2563 on OpenBazaar:master.

Coverage increased (+0.05%) to 62.356% when pulling 4a07b6811d95df999949b3a894053fae840fddb4 on cpacia:master into 5f728b384572e5dc7567aa6b748227b8acce2563 on OpenBazaar:master.

Coverage decreased (-0.9%) to 61.36% when pulling ee09c7c37919d17a2b03813b2bac2d3f6ad4f83d on cpacia:master into 5f728b384572e5dc7567aa6b748227b8acce2563 on OpenBazaar:master.


New Region To Exclude for Pairoscope WGS

A region we have seen in both Exome and WGS/WGL samples that should be added to the exclusion list is:
chr8:143,243,200-143,244,000 in MAFA region.
We also might have to extend the one we already have:
from chr8:143,844,450-143,844,750 to chr8:143,844,450-143,844,760

Fixed in the tumorOnly workflows for high coverage capture calling but needs to be fixed for the main version


chore(sdk): add automations codeowners

Description

Fixes WB-NNNNN
Fixes #NNNN

What does the PR do? Include a concise description of the PR contents.

[ ] I updated CHANGELOG.unreleased.md, or it's not applicable

Testing
How was this PR tested?

#8936
#8896
#8935
#8901
#9107  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


[Mention The Feature]:

Is your feature request related to a problem? Please describe.
the UI of this page is little messed up
Describe the solution you'd like.
i can fix the UI + add a dark mode feature.
@aditya-bhaumik please do consider this issue.
Show us the magic with screenshots

Checklist

[X] I have checked the Existing Issues
[X] I have read the Contributing Guidelines
[X] I want to work on this issue. (optional)

this issue has been solved, so we are closing this issue.


add config cdn for next/image

Codecov Report

Merging #19 (0163ae0) into main (d8fd9b5) will increase coverage by 0.27%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main      #19      +/-   ##
==========================================
+ Coverage   32.52%   32.80%   +0.27%
==========================================
  Files          13       13
  Lines         123      125       +2
  Branches       31       31
==========================================
+ Hits           40       41       +1
- Misses         78       79       +1
  Partials        5        5

Impacted Files
Coverage Δ

src/components/Footer.tsx
100.00% <ø> (ø)

src/pages/index.tsx
75.00% <50.00%> (-25.00%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8fd9b5...0163ae0. Read the comment docs.


Update build script test to have better isolation

when running in a dev environment.
Also ignore Pester test result files.

PR Summary

PR Context

I am trying to figure out why the CI is not working


Make getFile available in DSL scripts

Description
All properties of StructurizrDslPluginContext are available within a !script.
I need to access the getFile() property as well, because my script relies on files that are located in the same directory as the workspace.dsl. Otherwise there is no chance for the script to know the directory the DSL file is located in.
TLDR: Request is to make the workspace file property of  StructurizrDslPluginContext available to scripts
Priority
Low
Resolution
I have no budget, please add this feature for free
More information
No response

Once released, you'll be able to do context.getDslFile().


Unable to add Metadata to index

When trying to add metadata to an index, either using a list of metadata dicts or a mapping of uid to metadata dict (shown below), it always produces a key error.
Example:
RAG = RAGMultiModalModel.from_pretrained("vidore/colpali", device='cuda:2', verbose=1)

# contains 29 1-page pdfs
files = glob(os.path.join('/dataset_pdf', '*.pdf'))

# generate simple unique ids
uids = list(range(len(files)))

# get the file names
report_ids = [file.split('/')[-1].split('.pdf')[0] for file in files]

metadata = {uids[i]: {'file_name':report_ids[i]} for i in range(len(uids))}

RAG.index(
    input_path='dataset_pdf',
    index_name='Documents', # index will be saved at index_root/index_name/
    doc_ids=uids,
    store_collection_with_index=True,
    overwrite=True,
    metadata=metadata,

)

This produces the following error:
report_ids = [file.split('/')[-1].split('.pdf')[0] for file in files]
metadata = {uids[i]: {'file_name':report_ids[i]} for i in range(len(uids))}
--> RAG.index(
input_path='dataset_pdf',
index_name='Documents', # index will be saved at index_root/index_name/
doc_ids=uids,
store_collection_with_index=True,
overwrite=True,
metadata=metadata,
)

File ~/miniconda3/envs/rag/lib/python3.9/site-packages/byaldi/RAGModel.py:111, in RAGMultiModalModel.index(self, input_path, index_name, doc_ids, store_collection_with_index, overwrite, metadata)
def index(
 self,
 input_path: Union[str, Path],
   (...)
...
-->  current_metadata = metadata[i] if metadata else None
 if current_doc_id in self.doc_ids:
 raise ValueError(f"Document ID {current_doc_id} already exists in the index")

KeyError: 0

Removing metadata solves this problem, however, it should be ok based on the metadata docstring from RAGMultiModalModel

Hey

Removing metadata solves this problem, however, it should be ok based on the metadata docstring from RAGMultiModalModel

It should be yes, this is my mistake. The docstring is about the ideal scenario, which I had to rollback to not hold back the release because of slight metadata issues. Currently it only works with a list of len(documents), but the proper dictionary support will be re-added. Flagging this as a bug as you're expecting a behaviour that should be here.

Hi @NMVRodrigues, @bclavie
I also had this issue. Each time I tried to adding the metadata field it kept giving me this error:
`-->  current_metadata = metadata[i] if metadata else None
if current_doc_id in self.doc_ids:
raise ValueError(f"Document ID {current_doc_id} already exists in the index")
KeyError: 0
`
As a workaround for now sending each metadata as a list worked for me. So originally my metadata looked something like this:
items = [{"filename": "file1.pdf"}, {"filename": "file2.pdf"}, {"filename": "file3.pdf"}]
I changed to:
items = [[{"filename": "file1.pdf"}], [{"filename": "file2.pdf"}], [{"filename": "file3.pdf"}]]
Notice how I converted each metadata to a dictionary inside a list. It worked for me. Obviously, this is a work around and expectation would be the ideal situation as in the docs.
Let me know if you think there is something wrong with this.

Curious on the status for this. I noticed there was a fix out but hasn't been merged yet.

I've been out for various reasons (vacation/illness), but there'll be a byaldi release tomorrow or Monday (in the worst case) incorporating lots of tiny fixes!

Having different error message, but I think is related to metadata. When is validating the length of the documents against the metadata length, is using the dictionary instead of the list.
Number of metadata entries (2) does   [914071997.py](file:///tmp/ipykernel_5199/914071997.py):[78](file:///tmp/ipykernel_5199/914071997.py#78)
not match number of input items (1)


Load rule files form directories

I didn't enforce a .rules ending for backwards-compatibility.
@beorn7

:+1:

:+1:


Feature request: teach docopt to generate a manpage template

The programmer is already putting work into writing the docstring containing the options and usage patterns that is passed to docopt, so why not remove some redundancy and spare him from having to write a manpage from scratch, by teaching docopt to generate a manpage template?

@tobimensch too bad I can't rely on that, while docopt seems to be actively maintained, cli2man looks abandoned (pull request 6 months old) and it's not even published to pypi.python.org
@keleshev I strongly suggest you to consider merging cli2man into docopt

@pitchforks ... I'll look into the pull request again, but if I remember correctly it wasn't anything major that was broken, one request was to remove white space... not that that's bad, but not really high priority. But I'll look into it again. It's not abonened, I created actual manuals with it, look at the manual from my project on github termsql for an example, which I think turned out really great. If there's interest, especially if people post bug reports etc. I'll definitely try to provide fixes. The code might need some cleaning up (and I don't mean reformating or fixing white space), if someone wants to help with that for example, I would appreciate pull requests/patches. The most difficult problems of generating manpages are basically solved, it's at least 90% there. The rest 10% are basically cleaning up, fixing bugs, more testing against example programs and small incremental improvements.

I've taken a different approach to this. Literally started hacking on something today. If you start with a manpage, can you generate a docopt usage string?
Introducing:
https://github.com/ghostsquad/ronn2docopt
Obviously this is lacking polish, and really is a proof of concept at this point. One thing that I would love to see in docopt is the ability to include paging functionality like you'd see with git checkout --help. Using go-bindata would allow you to store both full help manpages in the executable as well as the docopt usage strings, and ideally there should be a way to switch between full help and minimal help/usage output.


More search ability

As the site grows ever larger, and people refer to characters I don't know ever more frequently, it'd be nice to have the ability to search for characters, templates, settings, subcontinuities, etc., especially by metadata.
For example, I could search for  "Theo" and check 'characters', 'templates' (or something) and get the Theo template plus all characters with Theo in the name. If I searched for "Opal", I'd get the Theo who is nicknamed Opal and also any other characters who might have metadata relating to Opal.

Ability to search for posts via titles too. (Sandboxes is getting large, apparently.)

Bump up "search for characters by screenname" so people with aesthetic preferences can make sure their screennames are unique.

Also "search for characters by name" so people know if they're going to be doing a character someone else already writes.

Helped by #291.

Still to do (?):

[ ] Search characters by alias
[ ] Search characters by setting
[ ] Search characters by facecast (separately from the Facecasts page, possibly doing partial matches)
[ ] Check author exists when searching
[ ] Check template belongs to author, filter dropdown to author's templates
[ ] Search templates
[ ] Search galleries
[ ] Search settings, when implemented?

http://alicorn.elcenia.com/board/viewtopic.php?f=12&t=459&start=2240#p33623
Intersections of multiple search things.

Also helped by 0fd71605909cb39192f62c74466202688add2a12

And #364


Handle none values properly and fix avTransport events

This is a combined PR for #465 and #466 because both change the same code.
It fixes #458, #448 and #453.

Coverage increased (+0.04%) to 59.941% when pulling 6ff4b60aec7a8c15a971714bbd9ed876fdfbf2d4 on ghcs27:handle_none_values into 43b1fadee59b70b9fdedfbc2b27e26739eb4c73a on SoCo:master.

let hope this get merged quickly
as we really need this fix in domogik

can please test if this works for you?

no this one is not working for me ...
title is always empty, with my changes only title was filled in

Coverage increased (+0.04%) to 59.941% when pulling 9db194b8d21ab335f246e946483f734f797e285c on ghcs27:handle_none_values into 43b1fadee59b70b9fdedfbc2b27e26739eb4c73a on SoCo:master.

Coverage increased (+0.04%) to 59.941% when pulling 5a7aadd48d70b3644ea5d058b612925a1a8feecc on ghcs27:handle_none_values into 43b1fadee59b70b9fdedfbc2b27e26739eb4c73a on SoCo:master.

realized my mistake: I checked for not title_elt instead of title_elt is None
Now it should work.

This one works

The flake8 complaints are not related to the code I have changed. Is there anything that's keeping this from getting merged?

Coverage increased (+1.2%) to 61.075% when pulling 5a7aadd48d70b3644ea5d058b612925a1a8feecc on ghcs27:handle_none_values into 43b1fadee59b70b9fdedfbc2b27e26739eb4c73a on SoCo:master.

Is there anything that's keeping this from getting merged?

Yes this is intended, as Google play service is not sending it ...
On Thu, Mar 16, 2017, 20:57 Kenneth Nielsen<EMAIL_ADDRESS>wrote:

@KennethNielsen commented on this pull request.
I think this looks sound. It's a shame that Sonos are so relaxed about
following spec, but I guess then we will have to fix it on our side.
In soco/data_structures.py
https://github.com/SoCo/SoCo/pull/467#discussion_r106515942:

     restricted = element.get('restricted', None)

   if restricted is None:

       raise DIDLMetadataError("Missing restricted attribute")

   restricted = True if restricted in [1, 'true', 'True'] else False

   restricted = False if restricted in [0, 'false', 'False'] else True

Didn't look through all the bug reports that this is supposed to fix yet.
But this changes the behaviour so that a missing restricted field will no
longer generate an error and will default to True. Is this the intended
behaviour?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/SoCo/SoCo/pull/467#pullrequestreview-27442640, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADiy8SWsrJBYXzU8IWR1SzNSd1x-szqlks5rmZQvgaJpZM4LcwiA
.

In that case I think this should be merged. I'll wait a few days to allow some of the bug reporters time to test it. If no testing turns up I'll merge it as is.

Should I squash the commits?

@ghcs27 that's your call. The commits such as they are, show the progression of the patch pertaining to actual functionality (except for the "too long line one", which is purely style). Whether you want to squash to get rid of that one commit is your call. Whether you want to squash more is really up to your view of the work. Commits should preferably contains "small isolated packages of work".
Do note however, if you do, you do not need to open a new PR for it. Just squach in your local branch and force push to the ghcs27:handle_none_values and make a comment here about it, that's enough.

OK, then I'll just keep the commits.

This is all good then. I will let it sit for a week in total from I commented in the other bugs asking for testing and then I'll merge it.

I think this can be merged now!

Coverage increased (+0.04%) to 60.75% when pulling ba3a8a61f23a83a4c35f721c46ca5c023186fc6b on ghcs27:handle_none_values into 2ff2e3f3806d020b5a60dd8c9a529fea1b644e49 on SoCo:master.

Coverage increased (+0.04%) to 60.75% when pulling ba3a8a61f23a83a4c35f721c46ca5c023186fc6b on ghcs27:handle_none_values into 2ff2e3f3806d020b5a60dd8c9a529fea1b644e49 on SoCo:master.

Is this pull request ready to be merged?  I'm trying to use this library in a project, but I connect to the Sonos speakers via Spotify connect.  Thanks!

Hello, this apparently fixes the #453 for me.
That said there is still an error afterwards since apparently the title attribute is missing, but the initial reported problem
DIDLMetadataError: Missing restricted attribute
has been fixed.
Can this be merged?

@KennethNielsen could you please merge?

@ghcs27 uhh, sorry about that. When I saw this pop up in my email queue again I had a thought. While I have no problem meriging this as is, I feel like events are forcing the events and data structures code into the wild west.  So 2 thoughts that I wanted to share/discuss before merging:
As far as I remember from reviewing it, the existing data structures code was fairly spec compliant. In general, this is a good thing, because it means that you have somewhere to check if the implementation is correct. So any changes to the code, that changes it away from being spec compliant, really should be:

Documenter. Can I get you to add very clear comments in the places where you are changing semantics away from the spec?
Tested. Ideally, although maybe a little too late now, we should collect test cases from the issues and add those to the test suites, so that we can ensure that future changes will still allow these non-compliant events through. I don't know if you feel up to that. My fear is, that if I merge, probably it will be a little more difficult to convince bug reporters to collect test cases. On the other hand, this has also been sitting around for so long, that it might also end up annoying people that it doesn't just get merged. I'll take your input on this.

Regards Kenneth

FWIW, I ws able to get it to crash using the sample event subscription code and playing from Spotify Connect (get on the same wifi as your sonos, and in the spotify app, select your sonos as a speaker option). Not sure how to turn that into a test case, or if that helps.

Hello Kenneth,
Instead of creating exceptions what about the possibility to let the event handler drop/ignore non-conformant events or flag them as broken. This could be made a configuration setting when you subscribe for events
strict .... Current behaviour
ignore .... Ignore non conformant events
relaxed ... Try to get as much information out of the xml and forward this with a "broken" flag.
In my specific case (spotify direct) I get 3 events for one song change so my idea is to ignore the content of the events and poll the player for the metadata instead.

@jake I could help you turn that into a test case, but I would need someone that has the time to write it up first, so let's see what happens with this discussion.

@maru-sama that isn't a bad idea although it would add a little extra complexity. Unfortunately, I think we are knee-deep in "relaxed" already, so the idea of "strict", as in spec compliant is probably lost. It might be possible to allow switching between "relaxed" and "ignore" with relatively little complexity. Could you post that idea as a separate issue?

@ghcs27 let me know if you think I'm being a pain here. If you prefer to have this merged quicker, since it's been sitting around for so long, then just address 1 and I will merge. Then we can try to make a separate issue about collecting non-compliant events.

I'd vote for getting this in instead of waiting for an ideal fix, since it fixes a pretty much fatal error, rendering master unusable in this scenario.

@jake yes, we always want the fix, but we also want maintainable code 2 years down the road ;)

true, but now the code is not use able for some services.
I think we should merge this, and create a new task issue to implement the
real and maintainable fix
On Wed, Nov 15, 2017 at 8:15 AM Kenneth Nielsen<EMAIL_ADDRESS>wrote:

@jake https://github.com/jake yes, we always want the fix, but we also
want maintainable code 2 years down the road ;)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/SoCo/SoCo/pull/467#issuecomment-344506158, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADiy8fAY7VmQoStLTeUOcAQIrtT-fbvAks5s2o98gaJpZM4LcwiA
.

Well this patch still does not fully fix the spotify direct issue because the title attribute is missing too. I mentioned this in one of my earlier comments. It "just" fixes the issue of the missing restricted tag.

@KennethNielsen I've added a warning label concerning the non-compliance to the spec, I hope it fits your expectation.
I won't be able to write the test soon, so please consider merging without it, but I'll get around to it sometime.
As for @maru-sama's suggestion, making it a separate issue at #536 is probably the best, as it does not only concern the changes in this PR
@maru-sama For the missing title attribute, please see #409.

@maru-sama For the missing title attribute, please see #409.

Hmm, the data_structures.py change in #409 appears very similar to the one in this PR? They actually conflict so both cannot be applied at the same time.
The problem is a "Missing title element" exception because title_elt is None and neither PR seems to handle that situation.

Ok. I will merge this and then we can sort out test cases and #409 afterwards.

@ghcs27 can you add a comment to #440 about the change

@Cereal2nd @erinxocon @maru-sama @jake @amelchio I would very much like help, in collecting these non-spec compliant events that make the code break, if any of you feel up to it. Please see #545 for instruction about how to collect them.

Also, please see #536 for a drafted solution to make the events code more resilient all over against non-spec compliant events.


Configure code formatter and syntax checker

Code formatting and syntax checking workflows are not running properly.
The workflows need to be fixed and set to run on more branches than just master

Resolved in pull request #39


Data source

Could you please tell me where to get the same data as your example? Thanks!

Hi,
Hope you are doing well! Did you find the data source?
I am getting the error "No such file or directory: '/home/carto/exe1/data/glue/WINOGRANDE/xl/WINOGRANDE/train.tsv'". Could anyone please help me on this one?
Thanks,
Bharani.


feat(31): add assertion for NonEmptyList type

implements assertions for NonEmptyList<A> type as specified in #31, the implemented assertions are:

shouldContain
shouldContainAll
shouldContainNull
shouldContainNoNulls
shouldContainOnlyNulls
shouldHaveDuplicates
shouldBeSingleElement
shouldBeSorted

Take a look @rcardin and let me know if something needs improvement :wink:

Thanks, man 🙏. I'll review it in the next few days.

Thanks for the feedback @rcardin, I will update the PR accordingly in the next few days as soon as I'm available :)


Uncaught TypeError: Cannot read property 'numberFormat' of undefined

I'm trying to change the location of PivotUi to "pt" but an error occurs in pivot.pt.coffee:11
ERROR:
Uncaught TypeError: Cannot read property 'numberFormat' of undefined
My code:
if (data) {
    var renderers = $.extend($.pivotUtilities.renderers, $.pivotUtilities.export_renderers);

    this.result = data.body;

    targetElement.pivotUI(data.body, {
      renderers: renderers,
    }, false, "pt");

head:

Yes, you have to load pivot.min.js before the local file :)

Thank you!


UNEXPECTED ERROR

Hello I' ve tried downloading a torrent file using the xbmc module and I got this error. I don' t understand the error my self nor did I find another issue like this. If this is a duplicate a will close it.
USER, HOSTNAME and IP are obviously masked.
{ addresses: [ 'IP' ],
query: [ '_xbmc-jsonrpc-h._tcp.local' ],
type:
[ { name: 'xbmc-jsonrpc-h',
protocol: 'tcp',
subtypes: [],
description: undefined } ],
txt: [ '' ],
port: 8080,
fullname: 'HOSTNAME._xbmc-jsonrpc-h._tcp.local',
host: 'HOSTNAME.local',
interfaceIndex: 0,
networkInterface: 'enp2s0' }
UNEXPECTED ERROR: If this is a bug in WebTorrent, report it!
OPEN AN ISSUE: https://github.com/feross/webtorrent/issues
DEBUG INFO: webtorrent-cli 1.2.0, webtorrent 0.93.3, node v5.7.0, linux x64, exit 1
undefined:1
SyntaxError: Unexpected end of input
at Object.parse (native)
at IncomingMessage. (/home/USER/.nvm/v5.7.0/lib/node_modules/webtorrent-cli/node_modules/nodebmc/device.js:35:31)
at emitNone (events.js:85:20)
at IncomingMessage.emit (events.js:179:7)
at endReadableNT (_stream_readable.js:913:12)
at _combinedTickCallback (node.js:377:13)
at process._tickCallback (node.js:401:11)

WebTorrent version: 1.2.0 (webtorrent 0.93.3)
Node.js version: v5.7.0
Browser name/version (if using WebTorrent in the browser):

Thanks for the report! This is actually a bug in nodebmc which webtorrent-cli depends on.
They have a call to JSON.parse() which they need to wrap in a try-catch block to prevent uncaught exceptions like this one. I sent them a PR: https://github.com/gtuk/nodebmc/pull/1

PR was merged, now waiting for it get published to npm.

The fixed version is on npm now. This should be frixed.


Change volume when rotating the mouse wheel over the Streamus icon

This is a great feature of the volume icon in Ubuntu and I would love it if it were implemented in Streamus too. I don't know if it is possible but I hope it is.

Hey,
I don't believe this is possible. There's no real event handlers for user-driven events when the UI isn't open unless it's a keyboard shortcut.
Sorrrry! Blame Chrome not Streamus. :p

https://developer.chrome.com/extensions/browserAction


Anacondo spyder application launch error

I couldn't open spyder all of a sudden someday, so I reinstall anacondo but still didn't work.
I downloaded anacondo windows python 3.10 version 64bits, and when I open anacondo and launch spyder it apperared "application launch error" and the description said:

C:\Users\USER\anaconda3\lib\site-packages\paramiko\transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
Traceback (most recent call last):
File "C:\Users\USER\anaconda3\Scripts\spyder-script.py", line 10, in
sys.exit(main())
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\app\start.py", line 252, in main
mainwindow.main(options, args)
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\app\mainwindow.py", line 1822, in main
mainwindow = create_window(MainWindow, app, splash, options, args)
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\app\utils.py", line 289, in create_window
main.setup()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\app\mainwindow.py", line 773, in setup
PLUGIN_REGISTRY.register_plugin(self, PluginClass,
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\api\plugin_registration\registry.py", line 342, in register_plugin
instance = self._instantiate_spyder5_plugin(
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\api\plugin_registration\registry.py", line 184, in _instantiate_spyder5_plugin
plugin_instance = PluginClass(main_window, configuration=CONF)
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\api\plugins\new_api.py", line 331, in init
container.setup()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\plugins\pythonpath\container.py", line 55, in setup
self._load_pythonpath()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\plugins\pythonpath\container.py", line 136, in _load_pythonpath
system_path = get_system_pythonpath()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\plugins\pythonpath\utils.py", line 31, in get_system_pythonpath
env = get_user_env()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\utils\environ.py", line 71, in get_user_env
env_dict = get_user_environment_variables()
File "C:\Users\USER\anaconda3\lib\site-packages\spyder\utils\environ.py", line 60, in get_user_environment_variables
res = stdout.decode().strip().split(os.linesep)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaa in position 832: invalid start byte

Hey @cmtata, thanks for reporting. This problem is fixed in our latest version available in Anaconda (5.4.3). Please update by closing Spyder, opening the Anaconda Prompt and running there the commands mentioned in our documentation.
Let us know if it works for you.

I use “ conda update spyder” at anaconda prompt, and it works! Thank you 🙏
On Thu, May 18, 2023 at 2:47 PM Carlos Cordoba @.***>
wrote:

Hey @cmtata https://github.com/cmtata, thanks for reporting. This
problem is fixed in our latest version available in Anaconda (5.4.3).
Please update by closing Spyder, opening the Anaconda Prompt and running
there the commands mentioned in our documentation
http://docs.spyder-ide.org/current/installation.html#updating-with-anaconda
.
Let us know if it works for you.
—
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/20941#issuecomment-1552561631,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A755A7IDIGOHNP6XEC3M7BLXGXAXBANCNFSM6AAAAAAYF72E64
.
You are receiving this because you were mentioned.Message ID:
@.***>


How do I know how much RAM I need to deploy a contract?

Before sending a transaction to create a contract how could I determine how much RAM I need?
At the moment, when I send the transaction the system throw an exception because I don’t have enough RAM, in the exception I can see the RAM needed: ‘...has insufficient ram; needs 39574 bytes...’, but I would like to know this data beforehand.
Thanks!

The memory you need is roughly the size of your compiled wasm file.

Thanks taokayan, but the size of the wasm file is a lot different from the RAM that exception tells me.
To avoid other issues, the contract that I'm trying to deploy in mainnet is the "Hello contract" delivered with EOS code. The size of Hello.wasm is 3042 bytes, and the exception returns 'account XXXX has insufficient ram; needs 40055 bytes ...'
I have deployed the contract on testnet and the spent RAM was nearly 40000 bytes.
Then, How could I know this data beforehand?

const static uint32_t   setcode_ram_bytes_multiplier       = 10;     ///< multiplier on contract size to account for multiple copies and cached compilation

   int64_t new_size  = code_size * config::setcode_ram_bytes_multiplier;

Also the abi takes up some room, but there is no multiplier on abi size and it is stored as packed bytes, so its impacts is small.
So roughly 10 * wasm size + packed abi size.


Memory leak in ol.control.FullScreen

Fixes #5110.
Register and unregister listener in setMap method.

Very elegant fix. Thanks @AvatharDG. Can you please submit a CLA so we can merge this?

No problem @ahocevar. CLA submitted.


Temp remove test_mtu_sized_frames - IBM4 debug

IBM4 setup has reveal an issue with MTU setting
and usage. Removing this test from running while
the debug is in progress to allow this other
provider to be added.
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date:

[x] README in the role
[x] Content of the docs/source is reflecting the changes

/approve


contract errors

I rolled back to an old version of the contract, restarted my local chain, make update
and I can now mint and claim again:
branch old-contract

eth_blockNumber (8)
eth_chainId
eth_estimateGas
eth_gasPrice
eth_getTransactionCount
eth_sendRawTransaction
  Contract call:       YourCollectible#claimToken
  Transaction:         0x129e77705c506408bce2aef987e674cbaa04072eb4514dc0f7951945e1893c5c
  From:                0x72df21b0bffdd667394721938016df8f25ddab83
  To:                  0x5fbdb2315678afecb367f032d93f642f64180aa3
  Value:               0 ETH
  Gas used:            112895 of 112895
  Block #6:            0x5d8999fc4beebd1180f8a64ea2f99ede3676596d93d4d4aa68acee919779f92f

eth_chainId
eth_getTransactionByHash
eth_blockNumber
eth_chainId
eth_blockNumber
eth_getLogs
eth_getBalance (2)
eth_accounts
eth_getBalance
eth_accounts
eth_call
  Contract call:       YourCollectible#balanceOf
  From:                0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266
  To:                  0x5fbdb2315678afecb367f032d93f642f64180aa3


Decimals and other binary bit sizes

How does this change with decimals and other binary bit sizes? Can they be added?

Do you mean support for e.g. half-float's or long doubles?
Larger floating-point numbers, i.e. numbers with a greater exponent range than doubles, would require enormously large tables. But I think implementations for half-floats would be quite easy to add.

Name
Common

binary16
Half precision

binary32
Single precision

binary64
Double precision

binary128
Quadruple precision

binary256
Octuple precision

bfloat16
Brain Floating Point

minifloats

TensorFloat-32

FP16

FP24

FP32

Extended precision formats

decimal32

decimal64

decimal128

Found in:

https://en.wikipedia.org/wiki/IEEE_754
https://en.wikipedia.org/wiki/Bfloat16_floating-point_format
https://en.wikipedia.org/wiki/Minifloat
https://medium.com/@moocaholic/fp64-fp32-fp16-bfloat16-tf32-and-other-members-of-the-zoo-a1ca7897d407

Can the table be properly generated than hardcoded?

Implementations for the minifloat formats can certainly be added. I think it wouldn't even require too much work (since most of the binary32 implementation can be reused).
The table for binary128 would ~2.5MB large. The table for binary256 would be much larger. (Both tables could probably be compressed, though) So binary80/128/256 are out of scope for this library and will certainly not be implemented. Generating the tables at run-time would require a big-integer implementation, which is also out of scope for this library.

I am looking to port this to Java. Is it possible to add a float/binary32 implementation?

I'm not sure what you are referring to.
There are already binary32 implementations of Ryu and Schufach. I'm not planning to implement binary32 versions of Grisu2/3 currently. (These are mostly for comparison with the other algorithms.) @jk-jeon has an implementation of Dragonbox which works for 32-bit floats here.
And there are already Java implementations of Ryu and Schubfach.
Note that porting the implementations in this repository to Java is not just a straightforward source code transformation. The implementations here all follow JavaScript's Number.ToString method: The output is always shortest. Java has slightly different requirements, where sometimes more digits than neccessary will have to be emited, and the Java implementations linked above both already do this correctly.

@jk-jeon version of DragonBox has a lot of meta programming hence it is difficult to translate into Java easily. I am only interested in Dragonbox.

I am also interested in simplified yet fast string parsing if this is also possible.

I am also interested in simplified yet fast string parsing if this is also possible.

You may find fast strtod/strtof implementations in ryu_32/64.h. There is also the more recent fast_float library, which you may want to check out.
Since I do not have any time in the near future to work on other floating-point representations, I'm closing this issue. But I'll drop a note here, if/when I get the time to work on this.

hello @abolz,
in comment #3 you stated: 'The table for binary128 would ~2.5MB large. The table for binary256 would be much larger. (Both tables could probably be compressed, though) So binary80/128/256 are out of scope for this library and will certainly not be implemented.'
you didn't say how big a table for bin-80 will be, a naive exponential interpolation might assume ~85k, if the trick of ryu to use a shrinked table at the cost of few more calculations is applicable 17k could be practicable?
I think / wish it could be possible to produce a working 80-bit implementation, would like that very much, but lack the skills to code myself ... why I'd like it? it's one of very few approaches to check the correctness of double calculations other than manual evaluation ...

hello @abolz,
thank you for your fast reply. still flat part of my learning curve ... the exponential behavior is oriented at the exponent size and that is 15 for 80-bit as well as 128-bit floats? ... :-(
the 'small table' approach of ryu could reduce that to ~300 MB?, still impracticable I think. Will try to dive into once I find some time to try to understand the basic math ...

You're welcome.
But maybe instead of focusing on these large floating-point formats, you may want to focus on the implementation of Ryu for some of the smaller floating-point formats. Like binary16. Or some toy format, which you may invent yourself. The advantage is that you're able to easily test all of the numbers in these formats.

hello @abolz,
'small formats' - yes, that may be an interesting field in nearby future, I have heard rumors that KI and big-data are going that direction reg. bandwith usage. Also for other usage small formats will become practical once they deliver less deviating results. I'm working on 'last bit accuracy' which changes ' = 0.6 + 0.3 ' -> 0.90000004 with floats to better behavior as well as ' = 0.2 + 0.1 ' -> 0.30000000000000004 with doubles.
But:

for the moment I need to continue from the point I am, and that is checking corrected double results with different calculations to prove them right.
as a somehow viable migration path I have to start from what is currently in use, and that is for spreadsheets 64-bit doubles.

hello @abolz,
two questions:

found the note of Ulf Adams about 'small tables' in d2s.c:
'// -DRYU_OPTIMIZE_SIZE Use smaller lookup tables. Instead of storing every
//     required power of 5, only store every 26th entry, and compute
//     intermediate values with a multiplication. This reduces the lookup table
//     size by about 10x (only one case, and only double) at the cost of some
//     performance. Currently requires MSVC intrinsics.'
thus I'd expect a factor 26 ( stored values ) or factor 10 ( mentioned ) shrinking -> ~75 or ~160kB, could that be manageable?

thinking about use of long doubles more for enhanced precision than for extremely small or big values ... would it be practicable to split / triage the long doubles in A.) those inside the double range from -/+( 5E-324 .. 1.7E308 ) and B.) the others outside that ranges, develop a fast algorithm which doesn't need extra exponent capa but only account the better precision of the significant, and call that for ( real life common ) values in ranges A.) while using the slow 'generic' algo as a fallback for ( IMHO rare ) occurrence of B.) figures? Near to try to code it by myself, but would need reasonable amount of help and Ulf Adams is too busy at the moment.

And - if I understood the graph in issue#2 correct that Schubfach is nearly twice as fast as ryu for short strings - it would be interesting to build such as a variant of Schubfach ...